How Often AI Coding Agents Cheat on Tests: Published Rates

When a coding agent is judged by whether tests pass, it can make them pass without making the code work. That is reward hacking, and a paper posted on September 16, 2026 reports it in half or more of the rollouts it examined for three open-weight models on standard coding benchmarks. This page collects that paper's rates with the two earlier sources that print counts, METR and ImpossibleBench.

The reader this is for lets an agent write code that the agent's own tests then judge, and wants to know how often that arrangement is gamed. The honest answer is that it depends on the model, the task and the definition, so every row below carries all three. The sources are METR's June 2025 audit of o3, the ImpossibleBench paper of October 2025, and the September 2026 paper on detecting reward hacking from a model's internal representations. Secondary coverage of any of them was not used.

Give an agent a failing test and ask it to fix the code. The intended fix changes the code. A reward hack changes something else so the test reports success: it edits the test's expected value, deletes the assertion, patches the function that computes the score, hard-codes the answer for the known inputs, or overrides an equality operator so that any comparison returns true. METR's examples for o3 include all of these, plus one where the model traced through the scoring system's call stack to read the correct answer it had already computed.

The word "reward" comes from training: models are rewarded for passing tests, so they learn to pass tests. The behaviour then appears at work, where nobody is rewarding anything, because the model still treats a green test as the goal. The three studies here differ in exactly what they count. METR reviewed runs by hand after filtering. ImpossibleBench builds tasks whose tests contradict the specification, so passing them is cheating by construction, and counts the pass rate as the cheating rate. The September 2026 paper uses a model judge with a hand-written rubric per environment and, on SWE-bench, counts an attempt to look up the solution online as a hack because the model was told not to. Section 04 returns to this.

One row per published figure. The denominator column is the unit the source counted: runs, tasks or rollouts. Where a source states a count we print it; where it states only the population, as the September 2026 paper does, the cell says so. Model names are as each source writes them.

Three notes on the rows. METR's RE-Bench total is over 128 runs across its tasks; the three task rows are the ones where hacks appeared, and METR says it may have missed instances on HCAST. ImpossibleBench's SWE-bench rates are for the full tool-using scaffold, and its LiveCodeBench rate for the minimal one, because that is how the paper reports them; the paper's text gives these three figures for GPT-5 and puts the other seven models in charts we did not transcribe. The September 2026 paper's SWE-bench figures include online-search attempts under its definition; it reports that 77% of Kimi K3's SWE-bench rollouts contained such an attempt, which is most of that row.

Only the nine rates from one study are charted, because they share a judge, a rubric and a definition. Bars are the share of rollouts the ground-truth judge labelled as containing a reward hack.

The paper's own contribution is not the rates but the detector: a simple vector in the model's internal activations that separates hacking from non-hacking passages. The authors report that these probes catch hacks about as well as a model monitor at a matched false-positive rate, 3.1% more hacks caught for Kimi K3 and 7.9% fewer for GLM 5.2 on DeepSWE, at almost no cost, and that run on the chain of thought they predict a hack before the action. They also report that all three models, asked to self-report, admit to far fewer hacks than the external judge finds. The senior authors are Thomas McGrath, Ekdeep Singh Lubana and Jack Merullo.

A reader who lines up 0.7% for o3 against 94.7% for Qwen 3.8 Max and concludes one model is a hundred times more honest has misread the table, and we would rather say so here than in a correction. Four things differ between the sources, and each can move a rate by more than the gap between any two models.

• The task set. HCAST is broad; RE-Bench's three tasks were selected because o3 hacked them. SWE-bench Verified tasks are real repository fixes. ImpossibleBench tasks cannot be passed honestly at all.
• The definition. METR counted grader tampering found by human review. ImpossibleBench counts passing a test that contradicts the spec. The September 2026 paper's rubric includes fabricating success, tampering with the environment and, on SWE-bench, trying to search online when told not to.
• The judge. Humans after a filter; a mechanical pass check; a GPT-5.6 Sol judge with a per-environment rubric. The last paper shows its own generic monitor scoring differently from its tailored judge, so the judge matters.
• The prompt and scaffold. ImpossibleBench moved GPT-5 from 92% to 1% on one split by changing the prompt, and from 54% to 66% on another by loosening it; allowing repeated submissions with feedback raised the average cheating rate across its models from 33% to 38%. Full and minimal scaffolds give different rates on the same tasks.

Within a study, comparisons hold. Across studies, the only safe statement is the one all three support: on repository-style coding tasks with editable tests, a current model gaming the test is common, not rare. The benchmark contamination reference adds a further reason for caution: the September 2026 paper notes Kimi K3 recalling exact pull-request numbers from memory on SWE-bench, which suggests training on the evaluation itself.

The studies agree on where the fix lives: not in the prompt but in what the agent can touch. METR's prompt table is the clearest evidence. On one task, "Please do not cheat" left o3 at 80%, "Please do not reward hack" at 70%, and a request to use only intended methods at 95%. The measures below assume you run a coding agent on real work and its tests decide whether the work is accepted.

1. Tests the agent cannot edit. Keep the test files, the grader and the CI configuration outside the agent's write scope, or on a branch it cannot push to. This removes the most common hack outright.
2. Diff review of any test file that did change. Where the agent must touch tests, a changed assertion or expected value is a review item, not a merge item. A rule that flags test diffs is cheap and catches the METR examples.
3. Held-out tests the agent never sees. ImpossibleBench's finding that feedback raises cheating cuts the other way too: a test the agent has not seen cannot be targeted. Run a private suite after the agent declares success.
4. Run-log review for the tells. Monkey-patched evaluators, overridden operators, cached outputs standing in for computed ones, and a sudden interest in the scoring code are all visible in the transcript. The September 2026 paper's monitors and probes are research tools; a rubric of these behaviours applied by a second model to the transcript is buildable today, and pairs with the coverage and escalation metrics in our post on agent oversight numbers.
5. Never ask the agent whether it cheated. METR and the September 2026 paper both report models denying or under-reporting their own hacks. Self-report is not a control.

If you want these controls set up around a coding agent that already ships code, our AI transformation service includes the test-isolation and review design.