An AI agent that passes a task in the demo may fail the same task tomorrow with nothing changed. On September 15, 2026 IBM Research put a number on how often. Running one agent five times on each of 168 tasks, it found that 77.4% of runs succeeded on average, but only 53.0% of tasks succeeded on all five runs. The gap between those figures, 24.4 percentage points, is the share of tasks the agent can sometimes do and sometimes cannot.
This post is for a founder or engineering lead who has watched an agent succeed and is deciding whether to ship it. It explains the two numbers in plain words, shows what IBM measured, and sets out how to run the same test on your own agent this week. The facts are from IBM Research's post on Hugging Face and its technical report on arXiv, submitted September 8, both read on September 22, 2026. It is one agent, one model and one benchmark; the finding is the gap, not the level.
- 01Average success and every-time success are different numbers.The average across runs, which IBM calls Mean@k, is what benchmarks report. The share of tasks passed on every run, Pass^k, is what a customer experiences when they ask twice. The second is never higher.
- 02On IBM's test the gap was 24.4 points, and 30 on hard tasks.A ReAct agent on GPT-4.1, five runs per task on AppWorld's 168 test_normal tasks, at temperature zero. The agent was capable and inconsistent at the same time.
- 03The flaky steps can be found without re-running the task.IBM's Consistency Analyzer resamples each decision in a recorded run, five completions per step, no environment replay and no correct answer needed.
- 04Guidelines from the agent's own runs halved the gap.Every-time success rose from 53.0% to 69.0% and the average did not fall. That is IBM's own result on its own benchmark; the method is open source.
01 — The definitionsTwo numbers, one agent
Take an agent, give it a list of tasks, and run every task five times. There are two honest ways to score it. The first is to count how many of the runs passed and divide by the number of runs. IBM calls that Mean@5, and it is what almost every benchmark reports as "accuracy". The second is to count how many tasks passed all five times and divide by the number of tasks. IBM writes that as Pass^5, and it is the question a user is really asking: if I ask this again, will it work again?
The second number can never be higher than the first, and on IBM's test it was a lot lower. Nearly a quarter of the tasks were ones the agent could solve on some runs and not others, with nothing about the task changing in between. IBM names that difference the consistency gap. It is not the same as the familiar Pass@k from code-generation papers, which asks whether at least one of k tries succeeded; that is the optimistic reading, useful when you can check and retry. Pass^k is the pessimistic one, and it is the one that matters when the agent acts without a human checking.
This isn't a capability problem you fix with a bigger model. It's an orthogonal axis.IBM Research, 'Your Agent Aced the Task. Will It Do It Again?', September 15, 2026
02 — The causeWhy a demo's success rate flatters
Every decision an agent makes, which tool to call, what argument to pass, whether to retry, comes out of a probability distribution over the next tokens. IBM's explanation is about the shape of that distribution. When one option carries most of the probability, the same choice comes out run after run. When several options are close to tied, the choice is near a coin flip, and small platform-level noise, such as how requests are batched or how floating-point arithmetic rounds on the GPU, can tip it. A task is a chain of dozens of such decisions, so a small chance of flipping at each one compounds into a large chance that some run goes a different way.
The part that surprises people is that this survives the usual fixes. IBM's agent ran at temperature zero, so none of the variation came from ordinary sampling. Greedy decoding and a fixed seed decide how a distribution is turned into a token; they do nothing about the distribution itself, and on a hosted model the probabilities shift slightly from one request to the next. A demo shows you one draw. Production shows the customer every draw. That is the same reason we argued for replayable runs in our reference on agent run replay.
03 — The dataThe published figures
Everything below is IBM's own measurement of one agent: a ReAct loop on GPT-4.1, the 168 tasks of AppWorld's test_normal split, five fresh runs per task, with consistency guidelines generated from a single recorded run of each task. GPT-4.1 is not a current frontier model, which is beside the point; the finding is the size of the gap and how much of it a cheap intervention closed. It is research-run, one agent, one model, one benchmark.
Average success versus every-run success, before and after guidelines
IBM Research, September 15, 2026. ReAct agent on GPT-4.1, AppWorld test_normal, 168 tasks, five runs per task. Research-run.| Measurement | Change in Pass^5 | Note |
|---|---|---|
| Easy tasks | +12.2 points | Least room to gain |
| Medium tasks | +22.9 points | About 44% relative, IBM's figure |
| Hard tasks | +14.3 points | About 45% relative; the baseline gap on hard tasks was about 30 points |
| Similar task, same scenario | +13.0 points | Guidelines mined from one task applied to a related variant; three points below the same-task gain |
| Second model, gpt-oss-120b, same task | +6.0 points | From 10.1% to 16.1%; on similar tasks the gain was +8.7 points, larger than same-task |
04 — Your agentMeasure your own gap this week
None of this needs IBM's tooling. The measurement is a loop and a spreadsheet, and the diagnosis is five sampled completions at each step of a run you already have. Three steps, in the order that gives you a number soonest.
- Run every task in your test set five times and record pass or fail per run. Report both numbers side by side: the share of runs that passed and the share of tasks that passed every time. IBM’s advice is that even three runs will show a gap you did not know you had.
- Sort tasks by how many of the five runs passed. The ones on three or four out of five are your inconsistent set. Those are the tasks your customers will describe as "it works sometimes".
- Resample the decisions in one recorded run of each inconsistent task. At each step where the agent chose a tool or an argument, send the same context back to the model and ask for five completions. Steps where the five disagree are the flaky ones. This needs no re-run of the task, no live tools and no ground truth, which is what makes it usable on production traces.
If you already keep the traces our observability checklist asks for, step three is a script over data you have.
05 — The fixWhat to do about the flaky steps
Guidelines from the agent's own good runs
Turn each flaky step into a short written rule and put it in the agent's context on similar tasks. IBM's example rules are plain: count checkbox markers with a line-anchored pattern rather than a substring, and confirm you have the right note before acting on it. The rules target instability, not failure, so they cover steps the agent got right this time by luck.
Tighter tool contracts
Many near-tied decisions are the model choosing between two tools or two argument shapes that both look plausible. Remove the ambiguity: one tool per job, typed arguments, and an error message that says what to do instead.
Deterministic sub-steps
Where a step is really arithmetic, parsing or lookup, take it out of the model's hands entirely. A counting step done in code cannot flip. The model decides what to count; the code counts.
The order matters less than the measurement. Whichever fix you apply, run the five-times test again and check that the every-run number rose without the average falling. IBM treated that as a hard requirement: a change that improves consistency by giving up average accuracy has moved the unreliability, not removed it.
06 — The limitWhen five runs is not enough
Five runs is a measurement, not a guarantee. A task that passed five of five has given you five successes; it has not shown you it never fails. For an agent that moves money, changes a customer record or sends something a human cannot recall, the right questions are different: what happens on the failing run, and does anyone see it before it lands? That is a control question, covered in our post on published reward-hacking rates from the other direction, where the agent passes the check by the wrong route.
Our rule for clients is to set the run count by the cost of a wrong run. Read-only agents: five runs and a reported gap. Agents that write to internal systems: the same, plus an approval step on the inconsistent tasks until their gap closes. Agents that act on customers or money: a human or a deterministic check on every action, whatever the consistency number says. If you want help building that measurement into your release process, our AI transformation service does it as part of every agent build.
The Consistency Analyzer and the consistency-guideline generator are in the open-source ALTK-Evolve repository. The technical report, "Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course", is arXiv 2609.08832, submitted September 8, 2026. The post lists twelve IBM Research authors, led by Evelyn Duesterwald, Lilian Ngweta and Vatche Isahagian.
07 — Next stepThe average tells you the agent can; the gap tells you it will
Run your test set five times before the next release
Add one column to your evaluation: the share of tasks that passed every run. Report it next to the average. Then pick the tasks that passed three or four times out of five, resample their decisions to find the steps that flip, and write those steps down as rules or move them into code. It is a week of work, and it is the difference between an agent that demos well and one a customer can rely on.