Is Anyone Watching Your AI Agents? Anthropic's Three Numbers

On September 17, 2026 Anthropic published a set of measurements it says would let the public see how fast AI is being built inside frontier labs. One of the three is about oversight: whether anyone is watching the agents that now do research and engineering work, how quickly, and how often they step in. Anthropic reports about 30,000 agents running at any one time on its most-used internal platform, every action passing through an automated monitor, and one action in roughly 47,000 blocked.

The numbers are Anthropic's own, for one platform, and the post is written in support of a policy position on pacing AI development that we report here and do not judge. What makes the post useful to anyone outside a lab is the three metrics themselves. They are simple enough for a five-person team to compute for its own agents, and specific enough that a manager, an auditor or a customer can ask for them by name. This post defines them, tables Anthropic's figures with their scope, and shows how to produce the same three numbers for agents you run. Source throughout: Anthropic's post, read September 18, 2026.

The post is published under the Anthropic Institute banner and is co-authored by Marina Favaro and Phillie Wright. It proposes three measurements of how models are built, as a complement to the capability evaluations that show what models can do. Anthropic frames the release as transparency ahead of a possible industry agreement to slow down, the position its chief executive set out in an essay we covered in our post on the pacing debate. The company says it expects the numbers to shift if such coordination happened, and that it plans to embed outside evaluators with access comparable to its internal risk teams.

The first and third measures are about the lab. The second is the one this post is about, because the metric behind it applies to any organisation that lets software take actions on its behalf. The R&D index uses an automation scale from Epoch AI, a research group, which runs from no AI involvement to fully autonomous; Anthropic says Claude is not fully autonomous on any measured subset of its work, and that the share at or above "AI collaborates" is above 90%.

Anthropic describes two kinds of monitor. An online monitor checks each action before it runs and can block or redirect it; it is built for harm that happens fast and cannot be undone, the post's example being an agent copying model weights out of the company's systems. An offline monitor reads the record of what agents did after the fact, looking for slow-developing problems such as signs of misalignment. The three metrics are defined against those two monitors.

• Coverage is the share of an agent's actions that pass through a monitor, before execution for the online kind or after it for the offline kind. If your agent calls a tool and nothing looks at that call, coverage for that call is zero.
• Review latency is the time between an action and its review. It has two clocks: the automated check, which for an online monitor happens before the action, and the human look, which happens only for what the automation flags.
• Escalation rate is the share of agent activity that is blocked or redirected by an online monitor, or flagged for further review by an offline one. It is a rate with a denominator: blocked actions over all actions, or flagged transcripts over all transcripts.

The point of stating them this way is that none needs the monitor to be clever. Coverage is a question about plumbing. Latency is a timestamp difference. The escalation rate is a count over a count. A team can report all three honestly with a rules-based checker and a spreadsheet, and improve the checker later.

Every figure below is as Anthropic states it. The scope column is the sentence that must travel with the number: all of the oversight figures cover one internal platform, the one Anthropic says is its most used, and none has been independently reproduced. The post says the figures were also published in the company's August 2026 risk report.

Two design choices sit under those numbers, and both are worth copying. Every agent on the platform has an individual identity that persists across model upgrades, so every action and every record can be traced to one agent over time. And agents talk to each other through a shared, open messaging system rather than privately, with every message tied to its author and cross-referenced to transcripts, so the monitor can follow a conversation between agents. Anthropic says the identity design also helps an agent treat another agent's output as a claim to check rather than its own thought.

A team running a few agents with real permissions, a support agent that can refund, a coding agent that can merge, an operations agent that can email customers, can produce the same three numbers in a week. The recipe below assumes only that every tool call is logged with a timestamp, an agent identifier and an outcome, which our agent observability guide covers in detail.

Three practical notes. First, define an "action" once and keep it: Anthropic counts decisions, which for most teams maps to tool calls, not model turns. Second, the human clock only starts at a flag, so a low escalation rate with nobody looking at the flags is not oversight; the escalation design patterns we published earlier describe who should own that queue. Third, keep the blocked actions. OpenAI's six misalignment reports from September 16 were found in exactly this kind of record, and a block you cannot replay teaches you nothing.

Anthropic is explicit about several limits, and a reader citing the figures should carry them. The oversight numbers cover one internal platform, not every agent the company runs. They are self-reported; Anthropic says it has been doing this monitoring "for months, not years", that it cannot be certain it has a complete picture of all possible behaviours at this scale, and that it expects the science to change. The compute figure is one week. The R&D index is judged by Claude, and Anthropic reports that its judge matched human raters exactly 59% of the time, against 35% agreement between the humans themselves, with model and human within one level in 97% of cases.

The gap that matters most for a team copying the metrics is the one Anthropic does not fill: a false-negative rate. Coverage of 100% means every action was inspected, not that every bad action was caught. The post says developers should publish how often known misbehaviour is caught and have third parties test it, and notes that METR, an independent evaluation group, has red-teamed Anthropic's offline monitoring. It gives no catch rate. Until you have run a seeded test of your own checker, treat your escalation rate as a lower bound on what is happening, not a measure of it. If you want help standing up the logging and the checker for a first deployment, our AI transformation service starts there.