OSWorld vs Terminal-Bench: What Agent Scores Really Mean

OSWorld and Terminal-Bench are the two agent benchmarks most often quoted side by side in model launch coverage, and they do not measure the same thing: OSWorld scores an agent driving a real desktop — windows, menus, files, browsers — while Terminal-Bench scores an agent working at a shell prompt. Reading one as a proxy for the other is the first mistake. Reading either as a single stable number is the second, and it is now the more expensive one.

Both franchises have forked. “OSWorld” today refers to at least two live, non-comparable benchmarks: OSWorld-Verified, the 369-task original, and OSWorld 2.0, a 108-task long-horizon successor announced on June 26, 2026 at its own separate domain. Terminal-Bench is running 2.0 and 2.1 leaderboards concurrently while a third generation, Terminal-Bench 3, already exists at a different domain again. A press release that says “tops OSWorld” or “state of the art on Terminal-Bench” has, at minimum, four possible referents.

This guide separates the families, maps which published figure came from which surface, and lays out three tables you can use as a reference: a disambiguation card for the OSWorld name, a vendor-table-versus-public-leaderboard ledger, and a harness-controlled comparison showing the same model scoring differently on the same test depending only on the agent wrapped around it. Every score below is attributed to the exact place it was published.

The clearest way to hold these apart is by what the agent has to operate. OSWorld puts a model in front of a running operating system and asks it to complete work the way a person would — open the application, find the setting, edit the file, verify the result. Terminal-Bench puts a model at a shell prompt and asks it to complete engineering work through commands. Both are agentic; the skills they stress barely overlap.

OSWorld’s evaluation design is worth knowing because it is unusually strict. Rather than asking a language model or a human to judge the output, the benchmark ships 134 execution-based evaluation functions that inspect the live environment after the run — file state, application state, and other observable ground truth. That removes an entire category of grader drift, and it is one reason OSWorld numbers tend to be lower than intuition suggests.

One more distinction that press coverage almost always drops: OSWorld’s own methodology write-up separates entries into three categories — a General model with broad-purpose capabilities elicited through prompting, a Specialized model trained specifically to act as a computer-use agent, and an Agentic framework that orchestrates several models together, for example a planner paired with a grounder. A single “model X scores Y on OSWorld” headline collapses all three into one number that may not describe the same kind of system at all.

For teams already tracking coding evaluations, this is the same discipline we applied in our SWE-Bench and Terminal-Bench methodology guide — a benchmark is a measurement apparatus, and the apparatus is part of the result.

OSWorld has changed twice in ways that break comparison across the name. First, an OSWorld-Verified upgrade dated July 28, 2025 fixed community-reported broken examples, added AWS-hosted evaluation that cut a full run to under an hour, and refreshed the published results. Pre- and post-verification “OSWorld” scores are therefore not directly comparable to each other.

Second, and far more consequentially, OSWorld 2.0 was announced on June 26, 2026 as a separate successor benchmark hosted at its own distinct domain — not a version bump of the same task set. It is a different, much harder test that happens to share a brand. The table below is the disambiguation card we use internally before accepting any OSWorld claim.

The row that does the most work is the interaction-depth row. OSWorld 2.0’s own abstract states the comparison plainly: a typical OSWorld 1.0 task needed roughly 30 tool calls with Claude Opus 4.7 at max thinking, versus roughly 318 tool calls for an OSWorld 2.0 task. Recomputing that ratio from the two published figures gives about 10.6× — an order-of-magnitude jump in how long the agent has to hold a plan together. A benchmark that changes by 10× in interaction depth is a new test, whatever the badge says.

There is a second, subtler trap in the vendor-stated OSWorld-Verified row. The highest figure we located, 86.1 for Qwen3.8-Max, sits above the 72.36% human ceiling reported in OSWorld’s original paper. That is not automatically a contradiction — the ceiling was measured on the pre-Verified task set and the vendor figure is on the refreshed one — but it is exactly the sort of cross-version comparison that should stop a reader, not impress one.

This is the part of the story that most reporting flattens. In Anthropic’s Claude Opus 5 launch comparison table, published on July 24, 2026, a row labelled “OSWorld 2.0” lists 70.6 for Opus 5, 66.1 for Fable 5, 62.6 for GPT-5.6 Sol, and 55.7 for Opus 4.8. On the OSWorld team’s own public OSWorld 2.0 leaderboard, the single highest binary-completion score across every entry is 20.6%, and it belongs to Opus 4.8 running at max effort with batched tool calls. Opus 5, Fable 5 and GPT-5.6 Sol are not listed on that leaderboard at all.

Take the one model that appears in both places. Anthropic’s table reports 55.7 for Opus 4.8; the public leaderboard reports 20.6% binary completion for Opus 4.8. That is a gap of 35.1 points, or roughly 2.7× — for the same model name on a benchmark with the same name. Compare the vendor table’s top figure of 70.6 against the leaderboard’s overall ceiling of 20.6% and the ratio is roughly 3.4×.

We are not going to reconcile those numbers, and neither should you. The plausible explanations — a different step or tool-call budget, a task subset, a scoring mode closer to partial credit than binary completion, or a run not yet reflected on the public board — could not be confirmed from the published sources. What can be stated confidently is narrower and more useful: both figures exist, each is attributable to a specific publisher, and neither belongs in the same column as the other.

Read the Terminal-Bench block carefully, because it contains the week’s most commonly mangled comparison. OpenAI’s own figures for GPT-5.6 Sol on Terminal-Bench 2.1 are 88.8 and 91.9 at ultra. The highest-placed GPT-5.6 family entry on the independently verified board is a different variant — Terra, inside Codex, at max effort, 78.4% ± 1.3%. The honest statement is that the family’s verified result and the vendor’s headline were produced on different surfaces for different variants. The dishonest statement, and the common one, is to quote the vendor number as if it were the board position.

The same discipline applies to Qwen3.8-Max’s vendor table. The 86.6 Terminal-Bench 2.1 figure and the 86.1 OSWorld-Verified figure both come from Qwen’s own published comparison. They are internally consistent and worth knowing, and we cover the wider release in our breakdown of Qwen3.8-Max’s benchmark set. They are simply not leaderboard positions, and no amount of hedging language converts them into one.

The Terminal-Bench 2.1 leaderboard does something unusual and valuable: it tracks “Agent Org” and “Model Org” as separate columns. That structure exists precisely to stop the conflation of “the model won” with “the vendor’s own agent won” — the board’s third-placed entry, for instance, is an Anthropic model running inside an agent built by the Terminal-Bench team, not inside Claude Code.

Because several models appear on the board under more than one harness, you can hold the model constant and read the scaffolding effect directly. As far as we can tell, nobody has laid these pairs out side by side. Here they are.

Three model families, three consistent directions: the vendor’s own agent edges out the generic reference agent every time. The magnitude varies enormously — 3.4 points for Fable 5, 2.8 for Opus 4.7, 0.2 for Gemini 3.1 Pro — which is itself the interesting part. A 3.4-point swing is larger than the 0.7-point gap between the board’s first and second entries. In other words, on this benchmark the choice of scaffolding can matter more than the choice between two frontier models.

That has a direct operational consequence. If your team evaluates a model inside a harness that differs from the one used for its published score, expect the published score not to reproduce — and expect the direction of the miss to depend on how much the harness was tuned for that model. We walked through the same effect from the coding-benchmark side in our analysis of scaffolding versus model capability, and the conclusion is identical: the harness is a variable, not a constant.

Terminal-Bench 2.1 reports per-run evaluation cost next to accuracy. These are the benchmark’s own published run costs — not API list prices, not subscription rates, and not an estimate of what the same work would cost in your stack. On that surface, the top two entries tell a story that accuracy alone hides completely.

The interpretation worth carrying forward is not “Fable 5 is cheaper.” Run costs on a fixed benchmark reflect the model, the harness, the effort setting, and how efficiently the agent uses its tool budget, all at once. What the numbers establish is that efficiency is a first-class dimension of agent performance and that it is already being published — and that a leaderboard summarised as a ranked list of percentages throws away half the available signal.

The same pattern shows up on OSWorld 2.0 in token terms rather than dollars. The leaderboard’s top entry, Opus 4.8 at max effort, spends roughly 244K tokens per task; GPT-5.5’s entry lands in the 13–14% binary-completion range while spending roughly 38.6K — about 6.3× fewer tokens for something over half the completion rate. Whether that trade is good depends entirely on whether you are optimising for completed workflows or for cost per attempt.

OSWorld 2.0’s most useful contribution is not its headline number. It is the shape of the decline. The project publishes binary completion broken out by how long a human takes to do the task, and the curve does not gently taper — it falls off a cliff.

Above roughly 163 minutes of human task time, binary completion goes to zero for every model plotted. Not low — zero. That is a structural limit rather than a capability gradient, and it is the single most decision-relevant fact in either benchmark family. If your intended workflow is a multi-hour, many-step process with state that has to survive the whole run, no published score in this article supports expecting it to complete unattended today.

The project’s own diagnosis of why is more precise than “the tasks are hard.” Agents are not failing at GUI mechanics or at writing code. They lose track of constraints stated earlier, miss information that arrives partway through, guess instead of asking, and skip verification — and the project describes them as struggling most when a task hinges on hidden state they have to recover.

Every item in that list is an architectural problem you can design around rather than a model limitation you have to wait out. Shorter task decomposition keeps runs inside the horizon where completion rates are non-trivial. Explicit constraint restatement between steps counters the drift. A mandatory verification step, and a checkpoint where the agent is permitted to ask rather than assume, addresses two more. The benchmark is describing a scaffolding specification, not just reporting a failure.

That is also our forward read on the next two quarters. The interesting movement will come from harness design and orchestration rather than from raw model gains — the harness-controlled table in section 04 shows the scaffolding is already worth more points than the gap between the two best models on the board. Expect the leaderboards to reflect that by increasingly reporting agent and model as separate, jointly credited entries, the way Terminal-Bench 2.1 already does. Teams that build their own evaluation harness against their own workflows will read these boards better than teams reading the headline percentage, which is exactly how we structure our AI and digital transformation engagements.

Terminal-Bench has the same naming problem as OSWorld, arriving from a different direction. Version 2.1 is explicitly framed by the project as an improved version of Terminal-Bench 2.0, inspired by Z.ai’s Terminal-Bench 2.0 Verified — meaning the point release is not a drop-in comparison to its predecessor without checking what changed. Meanwhile a Terminal-Bench 3 exists, hosted at a separate domain again and described as a benchmark meant to evolve with the frontier of agent work. We have not evaluated its methodology or scores; we note only that it is live, which is enough to make “tops Terminal-Bench” ambiguous.

Version sprawl is not a scandal; it is what happens when benchmarks saturate and the maintainers respond honestly. The problem is purely one of reporting hygiene. A score is only interpretable alongside its generation, its harness, its effort setting, and its publisher — and the moment any one of those is dropped from a summary, the number stops being evidence and becomes decoration.

There is one more caution worth carrying. Terminal-Bench results have not always survived independent rerun cleanly. In one documented case, a vendor’s own table showed a smaller model beating its larger sibling on Terminal-Bench 2.1, at 64.7% against 63.8%, while an independent rerun scored both at 55% — collapsing the ordering entirely. Reproducibility is not uniform across benchmark suites, and this one has visibly wobbled.

Everything above collapses into a short interrogation you can run in about a minute against any benchmark claim that crosses your desk.

1. Which benchmark generation? OSWorld-Verified or OSWorld 2.0. Terminal-Bench 2.0, 2.1 or 3. If the claim does not say, the claim is not usable.
2. Which surface published it? A vendor’s own launch or comparison table, or the benchmark project’s public leaderboard. These are different evidentiary categories and should never share a column.
3. Which harness ran it? The vendor’s own agent, a reference agent, or a third-party CLI. The Terminal-Bench 2.1 board shows this alone is worth up to 3.4 points on identical models.
4. Which effort or step budget? Max, xhigh, batched tool calls, and step caps all appear in leaderboard rows and all move the number materially.
5. What did the run cost? Where cost is published, read it. A 0.7-point advantage bought at roughly 3.7× the run cost is a different decision from a 0.7-point advantage at parity.

The same five questions transfer cleanly to adjacent claim types. They are the reason benchmark opacity extends to hallucination claims too, and the same opacity problem shows up in voice AI benchmarks. If you want a worked example of a board being read properly over time rather than at a single launch moment, see our walk through the live SWE-Bench leaderboard.

The practical version for engineering leaders is blunter. Use published benchmarks to shortlist, never to decide. Build a small harness that runs ten to twenty of your own real tasks, keep the effort settings and tool budgets identical across candidates, and record cost alongside completion. That internal board will disagree with the public ones, and yours is the one that describes your workload. It is also the artefact that makes a custom build engagement defensible when someone asks why you chose the model you chose.