The success rate on the agent dashboard was 84% last week and is 61% this morning. Two very different things produce that chart. One is a real regression: the agent handles tasks worse than it did. The other is a broken measurement: the agent is fine and the pipeline that scores it is not. Both cost the same to ignore and the wrong fix for each makes the other worse.
This post is the diagnostic for that morning. It gives the two tests that separate the cases, a decision tree built from them, a table of the five most common measurement breaks with the tell for each, what to do in the first hour, and the one recording practice that makes any of it possible. It names no incident at a named company, because the companies that have published postmortems of this kind describe the mechanism and we do not need their names to do the same.
- 01Shape first: a real regression has a slope and a spread; a measurement break is a vertical step at a boundary.Plot the metric per task and per segment, not just the aggregate. A capability loss degrades unevenly across task types over some period. A broken pipeline moves everything at once, at the timestamp of a deploy, a dependency bump or a config change.
- 02Corroboration second: find a signal the change could not have touched and see whether it moved.User-visible outcomes, a held-out eval on a pinned harness and model version, or billing volume. If the independent signal is flat, the agent did not get worse; the measurement did.
- 03Five breaks account for most false regressions: a tool schema, a renamed field, a logging drop, a harness upgrade, a sampling change.Each has a tell that can be checked in minutes from the raw traces, and each is fixed in the pipeline, not in the model.
- 04None of this works unless every eval result records the harness version and the model version it ran on.That one field turns a step on a chart into a query: which version boundary does the step sit on? Without it, every drop is a mystery.
01 — The problemTwo failures, one chart
An agent's score is the output of a pipeline: the agent runs, a trace is recorded, a grader reads the trace and the outcome, and a number is stored. A drop in the number can come from any stage. The agent can be worse. The trace can be malformed. The grader can be reading a field that no longer exists. The set of tasks being scored can have changed. On the aggregate chart these are one line going down.
The two failures demand opposite responses. A real regression means rolling back a model or prompt change and running the eval suite before the next deploy. A measurement break means fixing the pipeline and leaving the agent alone. Roll back the agent when the grader is broken and you lose a good change and keep a bad metric; fix the grader when the agent is broken and you have spent the first hour on the wrong thing while users notice.
Anthropic's engineering post on evals for agents describes the moment teams usually discover this: users report the agent feels worse after changes, and without evals the team cannot tell a real regression from noise. It separates capability evals, which ask whether the agent can do something at all, from regression evals, which ask whether it still handles everything it used to and should pass at close to 100%. This post is about what to do when that regression suite, or the production metric standing in for it, moves.
Teams can't distinguish real regressions from noise, automatically test changes against hundreds of scenarios before shipping, or measure improvements.Anthropic, Demystifying evals for AI agents, on building agents without an evaluation suite
02 — The testsThe two tests: shape and corroboration
The two failures share an aggregate but differ in almost everything else. Two tests expose the difference, and both can be run from data you already have.
A real regression
Degrades over a period rather than an instant, hits some task types harder than others, and shows a wider spread of scores per task. Per-segment lines diverge. The change rarely coincides exactly with a deploy that did not touch the model or prompt.
A measurement break
Moves in one vertical step at a timestamp. Every task type and every segment moves together, often to a round number such as zero. The timestamp lines up with a deploy, a dependency bump, a schema change or a config edit in the pipeline rather than in the agent.
Shape is suggestive, not conclusive, because a prompt change can also produce a step. That is what the second test is for. Corroboration means finding a signal that the suspected change could not have touched and asking whether it moved too. Three candidates are usually available. User-visible outcomes: tickets resolved, tasks accepted, code merged, complaints filed. A held-out evaluation run on a pinned harness and a pinned model version, which by construction did not receive any change. And billing or token volume, which tracks how much work the agent is doing regardless of how it is being scored.
If the independent signal moved with the metric, the agent got worse. If it stayed flat, the measurement broke. Google's Agent Development Kit evaluation documentation makes the point that agent behaviour is probabilistic, so a pass/fail assertion on one run is often unsuitable, and recommends evaluating both the trajectory of tool calls and the final response. The trajectory is a useful corroborating signal in its own right: a real regression changes what the agent does; a broken grader changes only what it is scored.
03 — The treeThe decision tree
Four branches, in order. Each is a question with a concrete check, and each ends in an action.
04 — The usual suspectsFive common measurement breaks
These five account for most false regressions we have seen in agent pipelines, ours included. Each has a tell that can be checked from raw traces and logs before anyone touches a model.
| Break | What happened | The tell |
|---|---|---|
| A changed tool schema | A tool the agent calls renamed a parameter or changed a return shape. The agent's calls now fail validation or return empty, and the grader scores the empty result as a failure. | Failures cluster on tasks that use one tool. Error logs show validation or parse errors from a single tool, starting at one timestamp. |
| A renamed field in the output or the trace | The agent's structured output or the trace it emits changed a key name. The grader reads the old name, gets nothing, and marks the task wrong. | Scores fall to near zero, not partway. A sample of raw outputs looks correct to a human. The grader's input, not its verdict, is what changed. |
| A logging or ingestion drop | Traces stop arriving, arrive late, or arrive truncated. The metric is computed over the traces that made it, which are not a random sample. | The denominator moved. Task volume in the dashboard does not match request volume at the gateway or the billing export for the same window. |
| A harness upgrade | The evaluation harness or the agent scaffold was updated: a new version of the runner, a changed system prompt, a different tool-call format, a new default effort or temperature. | The step lines up with a dependency bump or a deploy that did not touch the model. A rerun on the pinned harness scores as before. |
| A sampling or population change | The set of tasks or traffic being measured changed: a new customer segment, a new task type routed in, a sampling rate altered, a filter removed. | Per-segment scores are flat; the aggregate moved because the mix moved. The new segment or task type first appears at the step. |
The harness upgrade deserves a second look, because it is the one that changes the agent's behaviour without changing the model. A new default effort, a reworded system prompt in the scaffold or a different tool-call format can move scores in either direction, and the metric is telling the truth about a change that nobody classified as an agent change. Our replay and reproducibility reference covers what has to be recorded for a run to be rerun exactly; the harness version is the field most often missing.
05 — In practiceThe first hour, and the one practice
In the first hour, do four things and change nothing. Freeze deploys to the agent and the pipeline. Plot the metric per segment and per task type against the deploy log to run the shape test. Pull one independent signal to run the corroboration test. Rerun the last passing task set on the pinned harness and model. By the end of the hour the tree has resolved to one branch, and the fix is either a rollback or a pipeline repair, not both.
Record the harness version and the model version with every evaluation result and every production trace, and keep one pinned harness and one pinned model version running a fixed task set as a control. The recording turns a step on a chart into a question about a version boundary. The control gives the corroboration test a signal that exists by design rather than one you have to find on the morning it matters. OpenAI's evaluation guide points at a cookbook for detecting prompt regressions as you iterate; the pinned control is the same idea kept running.
Two related posts cover the surrounding practice. Anthropic's proposed oversight metrics for agents, and how to measure your own, are in our post on coverage, latency and escalation. The case where an agent's score goes up for the wrong reason, which the same two tests also catch, is in our review of published reward-hacking rates. If you want the control run and the versioned recording set up before the next incident, that is part of what we do under AI transformation.
06 — ConclusionA falling score is a question about the pipeline before it is a verdict on the agent
Version every result, keep one pinned control running, and run shape then corroboration before you roll anything back
The agent and the thing measuring it fail in ways that look the same on one chart and demand opposite responses. Shape tells you where to look first; corroboration tells you which one it was; the pinned control gives corroboration a signal that is always there. All of it depends on one field, the harness and model version stored beside every result. Add that field this week, and the next drop is a query instead of a mystery.