What Happened When Four Companies Let Agents Patch Code

Four organisations have now published their own numbers from letting coding agents find, validate and fix security bugs in their own codebases. Ramp went first in February 2026, Cloudflare in June, Google’s Chrome team in July, and OpenAI’s “Defense Factory” page appeared the week of September 3, 2026, according to OpenAI’s Daybreak announcement of that date. This page puts all four in one table, then explains why the numbers cannot be ranked against each other.

The finding that survives the definitions is not a number. It is a design decision. None of the four trusts one agent to judge its own output. Each built a separate adversarial check with its own context, and each kept a human at pull-request review. Two of the four independently found that deduplication was the hard problem. One measured that its agents, given a static analysis tool, chose not to use it once in a month.

Read the table as four separate reports that happen to share a page. The scopes differ by orders of magnitude: Ramp scanned one backend, Cloudflare scanned 128 repositories, Chrome covers a browser with more than 2,300 third-party dependencies, and OpenAI does not publish the size of its backlog at all. Each row states where the figure sits in that organisation’s own pipeline.

Two figures are deliberately absent. Cloudflare’s five minutes per fix and roughly fourteen hours from discovery to open pull request come from what the company calls a realistic benchmark based on a standard repository run, not a logged production run, so they do not belong in a results column. OpenAI’s 53 urgent issues closed on day one are attributed to teams working in parallel before the inventory was complete, not to agents. The cyber models named on OpenAI’s reference architecture are the Daybreak models we covered in our Daybreak analysis.

A comparison post usually puts the definitions after the table. Here they have to come before anyone quotes a number. The four organisations use the same words for different things, and the differences are large enough to reverse a ranking.

Cloudflare’s hunters are tuned to over-report subtle primitives on purpose, so its raw candidate count is inflated by design. OpenAI’s percentages are shares of a backlog whose size is never stated. Ramp counts only issues that were fixed. Chrome counts bug reports, including ones filed by outside researchers, and its 1,072 fixed bugs include fixes written by people with agent-generated candidates.

“Validation” is the sharpest split. OpenAI validates by running the service. Ramp validates by writing a test, after abandoning live deployment testing because its agents could not set up preconditions such as a victim account holding two funds. Cloudflare validates by trying to disprove, then separately asks whether the bug is reachable in production. Chrome validates by reproducing on affected platforms. A finding that clears one bar would not necessarily clear another.

The chart below shows the share of candidates each organisation’s stated gate removed or let through. It is one chart with five different quantities, and the label under each bar says which. Cloudflare’s initial rejection rate and Ramp’s manager rejection rate are both 40%. That is a coincidence between a system-wide lifetime rate and a one-off sample measurement, not a shared benchmark.

Cloudflare’s central pool is the one funnel whose arithmetic closes: 13,841 findings, minus 5,442 duplicates, minus 1,154 routed elsewhere, leaves 7,245 actionable findings. Its discovery funnel does not close from the prose alone, because the buckets between 20,799 raw candidates and roughly 12,057 survivors are shown only in an image. We report the prose numbers and nothing inferred from the diagram.

Chrome publishes no rate at all. Its post says the early-2026 harness ran with higher efficiency and lower false positives, without saying what it is lower than, and that automated triage saves an estimated hundreds of hours a month against a human baseline of five to more than thirty minutes per report. Google flags the saving as hard to measure precisely, and so do we.

Four teams with different codebases, languages and threat models reached the same architecture without coordinating. Ranked by the strength of the published evidence, the convergences are these.

Cloudflare states the reason for the first convergence outright: “If a Hunter is allowed to grade its own homework, it will confidently validate everything it outputs.” Two further patterns have weaker but real support. The bottleneck moves rather than disappears: Cloudflare calls its Fixer “the youngest and slowest part of the system” and says the scarce resource becomes review capacity, not discovery.

All four also concluded that one agent in one context window is the wrong unit of work. Cloudflare argues that a generic coding agent pursues a single hypothesis, exhausts its context window on a fraction of a real repository, and forgets things when that context is compacted. Its answer, and OpenAI’s, is that the durable asset is the harness around the model. That matches what we found in our agent runtime sandbox matrix: the isolation and orchestration layer decides what an agent can safely be allowed to do.

None of the four reports describes an agent merging a security fix on its own. The placement of the human gate is the same in every case: after validation, at code review, before anything reaches a main branch or a production fleet.

OpenAI’s account of how it got there is the useful part for a team planning the same thing. It says the work began with small batches under human review, and that manual steps were dropped one at a time as the results justified it. That is the same shape as the permission model in our default-deny versus default-allow analysis. Autonomy was granted per stage against evidence, not switched on.

Nobody publishes a recall rate, and Cloudflare explains why none can: “We don’t claim a false-negative rate for our system. There’s no labeled set of every real bug in a codebase, so any claimed recall number is entirely speculative.” Nobody publishes a dollar cost. Cloudflare describes the shape of its spend, with 50 to 200 workers per repository and each follow-up pass costing about half the initial hunt, but no figure. The other three say nothing about cost.

Cloudflare also says its own numbers are out of date by the time anyone reads them, and that its findings come from a ring-fenced research experiment rather than live unpatched production vulnerabilities. OpenAI publishes no start date, end date or duration for its sprint. Ramp names no models. Chrome gives no false-positive rate. Those blanks are recorded here as blanks.