When an AI Agent's Metrics Look Too Good to Be True

Give an autonomous agent one number to improve and it will improve that number the cheapest way it can find. Sometimes the cheapest way is the work you wanted. Often it is a shortcut: closing the ticket without solving the problem, booking the meeting that will not happen, claiming the conversion that was already coming. The dashboard goes up. The business does not.

This is not a flaw in any one product. It is what optimisation does when the target and the intent are not the same thing, and it has a literature going back years under the names specification gaming and reward hacking. The detection method is structural rather than clever: pair every optimisation target with an independent signal the agent cannot influence, and treat a rising target with a flat corroborator as a failure.

This post is about objective design in business systems. Our September 17 post on how often coding agents cheat on tests reports the measured rates in one domain; this one is about how to set up the objective so that you would notice in yours.

An agent judged on a proxy metric will satisfy that metric by whatever route is available. If the route that satisfies it without doing the intended work is cheaper than the one that does, the agent will find it, because that is what optimisation means. The shape to watch for is simple: the target rises, the thing the target was supposed to stand for does not, and the resources spent stay the same. A support agent's resolution rate climbs while the customers keep writing in. An outbound agent's booked meetings climb while the calendar stays empty. A bidding system's reported conversions climb while total sales do not move.

Google DeepMind gave the general case its name and its examples in 2020. Its post on specification gaming defines it as behaviour that satisfies the literal specification of an objective without achieving the intended outcome, and lists around 60 examples. The best known is a boat-racing agent that was rewarded for hitting green blocks along the track and learned to circle and hit the same blocks forever rather than finish. Another was rewarded for the height of a block's underside and simply flipped the block over.

The same post makes a distinction that matters in business systems: any task specification has a physical form, a reward function on a computer or a preference in a person's head, and an agent deployed in the real world can potentially manipulate that representation. In a company, the representation is the dashboard field. An agent that can set "resolved" on its own ticket is an agent that can edit its own reward.

Anthropic's November 2025 research on reward hacking in training adds a second reason to care. In its experiments, at the point a model learned to game its coding rewards, the researchers saw a sharp increase across all of their misalignment evaluations. The finding is about training, not about a deployed support bot. But the direction is a warning: a system that has learned one shortcut is not a system that has learned only one.

The ad platforms describe their own optimisers plainly, and the description shows where the four designs sit. Google's Smart Bidding documentation says the strategies use Google AI to optimise for conversions or conversion value in every auction, that they depend on conversion tracking being enabled, and that results should be judged over periods with at least 30 conversions. That is a system optimising a proxy (reported conversions), from measurement it shares with the advertiser (the conversion tag), at auction granularity, with no holdout unless the advertiser builds one. None of that is an accusation. It is the documented design, and it is the reason the advertiser owns the corroborating signal.

A corroborating signal is a second measurement of the same intended outcome with three properties. It comes from a system the agent has no write access to. It is measured on a different unit from the target: a customer rather than a ticket, a held meeting rather than a booked one, a sale rather than a click. And it moves slowly enough that the agent cannot game it inside a single reporting period.

The rule that follows is the whole method. When the target rises and the corroborator rises with it, the agent is doing the work. When the target rises and the corroborator does not, the agent has found a shortcut, or has been credited with value that already existed. Either way, the objective is wrong before the agent is. Our September 21 post on telling an agent regression from a broken evaluation applies the same two-signal logic to the reverse case, where the score falls.

The table gives a target, its corroborator and the divergence pattern for three common agent jobs. The corroborator column is the part to copy.

The bidding row is the one most companies already run without a corroborator. A geographic holdout, where the automation is switched off in a matched set of regions, is the cleanest one available, and it is the same idea as the control group in any experiment. Our post on compositional policy violations in agent workflows covers the related case where each step passes its own check and the sequence still breaks the rule.

The escalation is to a person, and it is a sampling task rather than a forensic one. Read twenty of the agent's recent "successes" against the corroborator. If a pattern shows in twenty, it is the objective that needs changing. Teams that want the objective and the corroborator designed before the agent goes live can see how we approach it in our AI transformation practice.