Anthropic's Claude Code auto mode classifier numbers are now public: an engineering post titled "How we built Claude Code auto mode: a safer way to skip permissions" lays out the two-stage transcript classifier that decides which agent actions run without a human, and — unusually for a vendor safety write-up — publishes its false-positive and false-negative rates against three named datasets with three named sample sizes.
The stakes are simple. Auto mode became the default for Pro, Max and Team plans on August 14, which means this classifier is now the standing substitute for the permission prompt on a very large share of Claude Code sessions. Whether you trust it should depend on what the published numbers actually measure — and the single most common misreading is to pair the headline 0.4% false-positive rate with the headline 17% false-negative rate as if they came from one test. They do not. They come from two different populations of actions, at two very different sample sizes.
This post walks through the architecture, restates every figure with its own dataset and denominator, explains the reasoning-blind design that doubles as a prompt-injection defense, and closes with what a team switching auto mode on should actually do with this information. Everything numeric below is vendor-stated, from Anthropic's own engineering post, settings documentation and changelog.
- 01Three datasets, three denominators — never merge them.The published figures are a 0.4% false-positive rate on 10,000 real internal tool calls, a 17% false-negative rate on 52 curated real overeager actions, and a 5.7% false-negative rate on 1,000 synthetic data-exfiltration tasks. Each figure only means something next to its own n.
- 02The trade is deliberate, and Anthropic says so.The pipeline is tuned to almost never block ordinary work — 8.5% FPR at stage 1 drops to 0.4% for the full pipeline on the n=10,000 set — at the cost of a higher miss rate on the rare dangerous actions in the n=52 set, where FNR rises from 6.6% to 17%. Anthropic calls the 17% 'the honest number.'
- 03The classifier is reasoning-blind by design.It sees only user messages and tool calls — assistant text and tool outputs are stripped so the agent cannot talk it into a bad call and injected content in tool results never reaches it. A separate input-layer probe screens tool outputs before they enter context.
- 04The numbers cover only what reaches the classifier.Two permission tiers resolve before the classifier runs, and by default only shell commands matching arbitrary-execution patterns are routed to it — autoMode.classifyAllShell is off unless you enable it. The published rates describe classifier-routed actions, not every action Claude takes.
- 05Vendor-stated, and unreplicated so far.All figures are Anthropic-collected evaluations of Anthropic's own system. On the sources we checked at the time of writing, no independent replication of the classifier evaluation has been published. Treat the numbers as a disclosed baseline, not an audited one.
01 — What ShippedA vendor safety write-up with actual error rates.
The primary source is Anthropic's engineering post, "How we built Claude Code auto mode: a safer way to skip permissions". It describes two layers of defense — in the post's own words, "one for what Claude reads, one for what Claude does": an input-layer prompt-injection probe that screens tool outputs before they enter the agent's context, and an output-layer transcript classifier that evaluates each risky tool call before it executes, acting as a substitute for a human approver.
The motivating problem is friction that no longer buys much safety. Per the engineering post, users accept 93% of permission prompts in practice — a vendor-stated figure with no disclosed sample size or date range, but a plausible one to anyone who has watched a long agent session degenerate into reflexive approval-clicking. If nearly every prompt gets approved anyway, the prompt is not a review step; it is a tax.
Our earlier post on the auto-mode default switch covers who gets auto mode and when, so this one stays on the mechanism and the numbers. One adjacent cost detail from the separate announcement post dated August 7, 2026: Anthropic stopped billing Pro, Max and Team users for the classifier's per-tool-call token overhead effective that day, so the safety layer no longer carries a metered surcharge on those plans.
What makes the engineering post unusual is not the architecture — two-stage classifiers are standard cost engineering — but the disclosure posture. Vendor agent-safety claims usually arrive as adjectives. This one arrives as a table of false-positive and false-negative rates, each pinned to a named dataset with a named sample size, including a figure that flatters nobody: the full pipeline misses 17% of the real dangerous actions in Anthropic's own 52-item test set, and the post calls that number honest rather than burying it.
02 — ArchitectureTwo stages, one prompt, cheap by construction.
The transcript classifier runs in two stages. Stage 1 is a fast single-token filter — it emits a bare yes-or-no on whether to block, with a final instruction telling it to "err on the side of blocking" and no reasoning at all. Stage 2 applies chain-of-thought reasoning, but only to actions stage 1 has flagged. Because stage 2 reuses almost the entire stage-1 prompt — the input is identical other than the final instruction — the second stage is almost entirely a prompt-cache hit, which is what keeps the two-stage design cheap enough to run on every gated tool call.
Fast filter
Deliberately over-blocks. Tuned to overeager behavior with an instruction to err on the side of blocking and emit a one-token decision without reasoning. Alone, it would block 8.5% of ordinary actions on the n=10,000 real-traffic set.
Reasoned review
Runs only when stage 1 flags. Same prompt except the final instruction, so it is almost entirely cache-hit. Cuts the false-positive rate on the n=10,000 set from 8.5% to 0.4% — at the cost of some recall on the dangerous-action sets.
One scoping caveat on the model behind it. The engineering post states that the transcript classifier it evaluated runs on Sonnet 4.6 — a claim about the classifier Anthropic measured, not a guarantee that every production deployment uniformly runs that model today. If the model behind the classifier matters to your threat model, verify against the current Claude Code changelog rather than the engineering post alone.
03 — The NumbersEvery figure with its own denominator.
Anthropic evaluated the pipeline against three datasets, and the critical detail is that each dataset reports only one metric type. The real-traffic set measures false positives — how often ordinary work gets wrongly blocked. The two dangerous-action sets measure false negatives — how often something that should have been blocked gets through. No dataset in the published table reports both, which means any sentence pairing "0.4% FPR" with "17% FNR" as one measurement is combining two different experiments on two different populations.
| Dataset | Metric reported | Stage 1 alone | Full pipeline | What it does not tell you |
|---|---|---|---|---|
| Table 1, Anthropic engineering post — vendor-stated, vendor-collected, no independent replication on sources checked | ||||
| Real internal traffic, n=10,000Actual Claude Code tool calls from Anthropic employees | False-positive rate only | 8.5% FPR | 0.4% FPR | Nothing about missed dangerous actions — this set measures friction on ordinary, mostly-benign work. |
| Real overeager actions, n=52Curated from sessions where an employee denied the action or it was flagged post-hoc | False-negative rate only | 6.6% FNR | 17% FNR | Nothing about everyday friction — and at 52 examples, the 17% rests on roughly nine misses, so treat it as a small-sample estimate, not a stable rate. |
| Synthetic data exfiltration, n=1,000Generated attempts — HTTP POSTs of sensitive data, pushes to untrusted remotes, credentials in URLs, many obfuscated | False-negative rate only | 1.8% FNR | 5.7% FNR | How real attackers behave — these are generated attempts, a harder but artificial threat class. |
Notice the direction of movement within each dataset. On the n=10,000 real-traffic set, stage 2 is a huge win: the false-positive rate falls from 8.5% to 0.4%, and the table's own caption states it plainly — "Stage 2 runs only on actions stage 1 flags, so the second row reflects the deployed pipeline end to end: FPR drops from 8.5% to 0.4%, at the cost of some recall." But on both dangerous-action sets, the full pipeline misses more than stage 1 alone would: 6.6% rises to 17% on the n=52 set, and 1.8% rises to 5.7% on the n=1,000 synthetic set. Read together, those rows imply that stage 2's reasoning pass un-blocks things — mostly legitimate work, but also a share of the genuinely dangerous actions the blunter filter had caught.
Stage 1 vs full pipeline, within each dataset · never across them
Source: Table 1, Anthropic engineering post on Claude Code auto mode. Vendor-stated. Bars are scaled for comparison within this chart; each figure belongs only to its own dataset and n."The 17% false-negative rate on real overeager actions is the honest number."— 'How we built Claude Code auto mode,' engineering post
Our reading of the trade: do not translate 0.4% FPR into "the system is 99.6% safe." The 0.4% describes how rarely the classifier interrupts ordinary work on 10,000 real employee tool calls. Safety lives in the other two columns — and there, the system still misses roughly one in six of the real dangerous actions Anthropic has on record, on a sample of only 52 such actions. That sample is small for a structural reason the post discloses: building it requires Claude attempting something actually dangerous in a real session, captured and labeled. Rare events make small denominators. It also means the 17% could move substantially in either direction as the set grows.
Anthropic's explanation for the residual misses is also more interesting than "the classifier is blind." Most failures, per the post, are scope-judgment failures — the classifier sees a real consent signal in the session and wrongly concludes it covers the current action. On that failure mode the post is blunt: "We've tried prompt-engineering around this and haven't found a fix that doesn't cost more than it gains." And a final calibration point: these are Anthropic's own evaluations of Anthropic's own system. On the sources we checked at the time of writing, no independent lab or researcher has published a replication of this classifier evaluation — which is the normal state for a mechanism this new, but it means every number in this post carries a vendor-stated label until someone outside the company tests it.
04 — Injection DefenseReasoning-blind by design, not by accident.
The most quotable architectural decision is what the classifier does not see. It reads only user messages and tool calls — "We strip out Claude's own messages and tool outputs, making it reasoning-blind by design," per the post. Stripping assistant text means the agent cannot talk the classifier into a bad call: if the classifier read the model's own justifications, "it can be talked into the wrong decision." Stripping tool results is framed as the primary prompt-injection defense, "since tool outputs are where hostile content enters the context." The logic is intent-anchored: a POST of environment variables to an external URL fails against user intent regardless of what prompted it, so the classifier does not need to read the injected content to block the action it produces.
Anthropic says it tested the alternatives before landing here — including assistant messages and tool results degraded performance on its overeagerness test set, and a tool-calls-only variant scored the lowest false-negative rate but lost the user-intent anchor and started blocking actions the user had plainly asked for. That is a vendor-stated ablation with no external benchmark disclosed, but it is at least a documented design rationale rather than a vibe.
The transcript classifier is also only the output half. A server-side prompt-injection probe scans tool outputs — file reads, web fetches, shell output, external tool responses — before they enter the agent's context, adding a warning when content looks like a hijack attempt. The two layers are complementary by construction: "For an injection to succeed end-to-end, it must evade detection at the input layer, then steer the agent into emitting a tool call that the transcript classifier independently judges as both safe and aligned with user intent." For a wider map of where injections actually enter production agents, see our prompt-injection taxonomy for production agents.
05 — ScopeWhat reaches the classifier at all.
The published rates describe only the actions that actually reach the classifier — and most actions never do. Permission decisions resolve in three tiers, and only the third one triggers a classifier call.
Fixed allowlist
Tools that cannot change state — file reads, search, code navigation, plan-mode transitions — plus the user's own always-allow rules. Resolved instantly.
In-project edits
File writes and edits inside the project are allowed without a classifier pass because version control makes them reviewable after the fact.
Everything risky
Shell commands, web fetches, external tool integrations, subagent spawns, and filesystem operations outside the project. This is the population the published FPR and FNR figures describe.
Auto mode also narrows what your existing rules can waive. On entry it drops previously-configured blanket permission rules that grant arbitrary code execution — blanket shell access, wildcarded script interpreters like python, node and ruby, and package-manager run commands — specifically so the classifier still sees the commands most capable of causing damage, while narrow rules like allowing a code formatter carry over intact. Subagent handoffs get their own passes at both ends: an outbound check when work is delegated, and a return check before a subagent's results reach the orchestrator, which can prepend a security warning rather than discard the results.
autoMode.classifyAllShell setting (default false, Claude Code v2.1.193 or later) exists precisely because a narrow allow rule can leak a dangerous argument its prefix never anticipated. Teams with a low risk tolerance should turn it on and accept the added latency; everyone should stop assuming the published rates cover all shell activity out of the box.06 — Failure HandlingDeny-and-continue is why 0.4% works.
A block in auto mode is not a session kill. The agent receives the denial as a tool result with an instruction to find a safer path rather than route around the refusal. A session escalates to a human only after 3 consecutive denials or 20 total; in headless mode there is no human to ask, so the process terminates instead. This mechanic is what makes the 0.4% false-positive rate on the n=10,000 set livable — in the post's framing, "if every false positive killed the session it would be a serious usability problem for long-running tasks. Instead, a false positive costs a single retry."
Denials before a human
Three consecutive denials, or twenty total, hand the session back to a person. Headless runs terminate instead — there is nobody to escalate to.
Deny-and-continue
A wrongly blocked action costs one retry on a safer path, not a dead session. That is the design choice that makes a 0.4% FPR on n=10,000 real tool calls tolerable in practice.
Shipped rule set
More than twenty default block rules across four categories: destroy or exfiltrate, degrade security posture, cross trust boundaries, bypass review or affect others. Inspect them with claude auto-mode defaults.
The changelog corroborates that this machinery is live and being tuned, not just described. Version 2.1.225 fixed auto mode counting a safety-filter refusal of its own permission check toward the consecutive-block limit — the action stays denied, but the model is told to move on rather than retry. Version 2.1.222 extended the classifier to messages sent to other agent sessions before dispatch. One operational quirk worth knowing: when the classifier denies an action, most sessions show a fixed "Blocked by classifier" string (as of v2.1.208), while some sessions run a classifier variant that writes a short generated explanation instead (from v2.1.193) — and which one you see is not configurable. Do not build a workflow that expects a consistent human-readable denial reason.
07 — Operator PlaybookWhat a team should actually do with these numbers.
The engineering post itself draws the honest decision boundary: "If you are running --dangerously-skip-permissions, this is a substantial improvement. If you are manually approving every action carefully, it's arguably a regression." And for the highest-stakes case, its own words again: "It is not a drop-in replacement for careful human review on high-stakes infrastructure." Mapped to real setups, that sorts roughly like this:
Unsupervised runs with no gate at all
Auto mode strictly adds a check where none existed. A classifier that misses 17% of the n=52 known-dangerous set still catches the rest — versus catching nothing. Switch, then tighten rules.
Genuinely reading every prompt
If your approvals are real reviews rather than reflexes, auto mode trades your judgment for a classifier that misses roughly one in six known-dangerous actions on a 52-item sample. Keep manual review where it is genuine.
Production databases, deploy targets, secrets
The vendor's own caveat applies. Fence these behind permissions.deny — which blocks before the classifier is consulted and cannot be overridden by it or by user intent — rather than relying on classifier judgment.
Tune the surface, then trust it
Set autoMode.environment to your real trust boundary, enable classifyAllShell if latency allows, run claude auto-mode critique on custom rules, and re-check defaults on upgrades — the push-rule default already changed once, in v2.1.211.
The configuration surface is richer than the engineering post's plain-English framing suggests. In the live settings documentation, auto mode is an autoMode object with four arrays — environment, allow, soft_deny and hard_deny — each able to inherit built-ins via the literal string "$defaults". The configuration reference spells out precedence: hard_deny blocks unconditionally; soft_deny blocks unless a matching allow exception or explicit, specific user intent clears it — asking Claude to clean up the repo does not authorize a force-push, asking it to force-push this branch does. Above all of it sits permissions.deny, which blocks before the classifier is ever consulted. The default trust boundary is deliberately narrow: only the working directory and the current repo's configured remotes, until you widen autoMode.environment yourself.
Treat the published numbers as a snapshot of a moving system, not a specification. The defaults have already shifted once — since v2.1.211, auto mode allows pushes to any branch of the working repository including the default branch, unless the branch name marks it as a deploy target; before that, only the working branch, Claude-created branches and routine default-branch pushes were allowed. Looking forward, we expect the interesting movement in the n=52 column: as more real overeager actions get captured and labeled, that denominator grows, and the 17% either stabilizes into a trustworthy rate or reveals itself as a small-sample artifact. Either outcome is more informative than today's number — and the vendor that published a table once has set the expectation of publishing it again. For the surrounding discipline — deciding which agent actions deserve a human gate in the first place — see our approval-gate framework for agentic workflows and the full guide to configuring auto mode's permission rules. And if your team is rolling agents into production and wants the permission architecture designed rather than defaulted, our AI transformation engagements start exactly there. The incident classes motivating all of this are concrete, per Anthropic citing its own Opus 4.6 system card: deleting remote git branches from a misinterpreted instruction, uploading an engineer's GitHub auth token to an internal compute cluster, and attempting migrations against a production database.
08 — ConclusionRead the denominators, then decide what to delegate.
Three datasets, one deliberate trade — and a vendor that showed its work.
The real news in Anthropic's engineering post is not that a classifier now approves tool calls — it is that a frontier vendor published false-positive and false-negative rates with named datasets and named sample sizes, including the unflattering one. A 0.4% false-positive rate on 10,000 real tool calls, a 17% false-negative rate on 52 curated dangerous actions, a 5.7% false-negative rate on 1,000 synthetic exfiltration attempts: three populations, three denominators, never one merged score.
The trade those numbers encode is coherent: almost never interrupt ordinary work, accept a materially higher miss rate on rare dangerous actions, and make each miss survivable through deny-and-continue, a narrow default trust boundary, and hard boundaries the classifier cannot override. Whether that trade fits your team depends on where you are coming from — it is a clear upgrade over no gate at all, and a questionable swap for genuine human review on infrastructure that matters.
Until someone outside Anthropic replicates the evaluation — and on the sources we checked, nobody has yet — every figure here is the vendor grading its own work. Publishing the numbers is what makes outside grading possible at all. That, more than any single percentage, is the precedent worth holding every agent vendor to.