Terminal-Bench 4.0 changes the conditions behind an agent score. Before using an older result to choose a coding agent, identify which tasks and settings changed, then decide whether the evidence requires a new run, a new grade or no substantive change. A model name and a percentage are not enough.
The official v4.0.0 release was published on August 26, 2026. It identifies removed tasks and changes to existing ones. The useful buying question is which of your comparison rows remain supported, rather than whether the new version makes an agent look better.
- 01Match versions before comparing.Changed tasks or resources can alter a result without changing the model.
- 02Separate rerunning from regrading.A new verifier may use retained artifacts; a changed agent environment may require new work.
- 03Record actual settings.Defaults and prose summaries do not prove which overrides a submitted run used.
01 — Practical guideChoose between rerun, regrade and reuse
Treat an existing result as evidence produced by a specific experiment. Rerunning repeats the agent’s work. Regrading applies an appropriate verifier to retained outputs. Reuse carries a result forward when the relevant experiment has not materially changed. Each route needs a reason.
The benchmark maintainers describe these distinctions in their continuous-benchmark methodology . Apply the change classification to the affected tasks. A major dataset update does not by itself establish that every individual task needs the same treatment.
| Change | Candidate action | Evidence needed |
|---|---|---|
| Task input or execution environment changes | Rerun affected work | Exact new task, resources, agent settings and retained outputs. |
| Verifier changes without changing the agent task | Regrade if the artifacts are sufficient | Compatible final artifacts, verifier version and both grading outcomes. |
| Documentation-only correction | Reuse if behavior and grading are unchanged | Change review confirming no material effect on the experiment. |
| Task removed or denominator changed | Recompute the comparison on a stated set | Included task manifest and scoring policy. |
| Change cannot be classified | Leave the result unresolved | More information from the benchmark or run owner. |
02 — Practical guideInspect the executable configuration
A concrete example appears in the pinned v4.0.0 task for batched-eval-parity. Its README summary describes four hours and one CPU, while the changelog describes eight hours and two CPUs. The task configuration specifies an agent timeout of 28,800 seconds and two CPUs.
That is a documentation/configuration discrepancy observed by reading the pinned files, not a benchmark run. It shows why the source and revision matter. The configuration is more directly relevant to execution than a stale summary, but even it does not prove the settings of a particular submitted result. Overrides may still apply.
Ask for the actual run configuration: resource allocation, hard limits, timeout, concurrency, agent revision and retry policy. Keep the source of every field. If a provider reports its default rather than the run’s value, mark that difference instead of filling the gap with an assumption.
03 — Practical guideKeep resources in the comparison
Anthropic’s infrastructure study reports that resource settings changed results in its Terminal-Bench 2.0 experiments while the model, harness and tasks were held fixed. It distinguishes guaranteed resources from hard limits. This is vendor-run evidence about that experiment, not a universal correction factor for version 4.0.
A larger allowance can change what an agent has time to attempt, while a hard limit can determine whether a process survives. Record both when they matter. Do not infer that increasing every resource always improves the score or that one percentage adjustment makes unlike experiments comparable.
Our OSWorld and Terminal-Bench explainer covers what the benchmark families measure. This article adds the migration question: how to preserve a defensible comparison when the measurement system itself changes.
04 — Practical guideBuild a migration record instead of a mixed leaderboard
Give each old-to-new comparison a record with its own inclusion rules. If the new dataset removes tasks, a shared model label does not preserve the old denominator. A score is meaningful only with the task set and outcome accounting that produced it.
| Record | Include | Reason |
|---|---|---|
| Dataset identity | Old and target versions, immutable references and task manifests | Makes additions, removals and changed tasks visible. |
| Agent identity | Harness revision, model identifier, reasoning and retry settings | Distinguishes a model change from a system change. |
| Execution | Resource allocations, hard limits, timeout and actual overrides | Prevents unlike environments from appearing equivalent. |
| Verification | Verifier revision, available artifacts and migration action | Explains whether the result was rerun, regraded or reused. |
| Outcome | Task denominator, attempts, success, timeout and error handling | Preserves the publisher’s scoring policy. |
| Provenance | Run owner, date, original record and measured costs if supplied | Separates vendor reports from independent reproduction. |
05 — Practical guideAsk for the evidence that changes the buying decision
Suppose a supplier shows an older result beside a version-4 score from another agent. Ask for the common task set, each run’s configuration and the policy used to carry results forward. If those are unavailable, treat the comparison as incomplete. You can still investigate both products without pretending their numbers are directly comparable.
The vendor benchmark reproducibility audit provides a disclosure baseline. The broader SWE-Bench and Terminal-Bench guide explains their differing tasks. Neither source should be used to manufacture a conversion between benchmark versions.
Avoid importing current leaderboard percentages into a historical comparison without a dated export. A present-day board can change after a submission correction or new run. Retain the original record that supported the decision, including unsuccessful attempts and missing fields.
06 — Practical guideKeep the migration narrow enough to audit
Start with the result you actually use to shortlist an agent. Classify its relevant changes and request only the missing evidence. If a new run is needed, freeze its task set and settings before comparing it with another system. If regrading is possible, retain the old artifacts and both verifier outputs.
No benchmark tasks were executed for this article, and no model ranking is inferred from the release. The migration table is a proposed editorial method. For help connecting evaluation evidence to a working application, our AI transformation service can scope an evaluation around the tasks that matter to your team.
Evidence and scope
- As-of date
- September 12, 2026. September 9 is the editorial allocation; this research was reviewed later.
- Method
- Primary release documentation and pinned task files were inspected. The result-migration table is a proposed method for buyers; it is not a live leaderboard.
- Limitations
- No benchmark runs, model comparisons or cost measurements were performed. Current methodology pages provide context; release-specific claims use the pinned August 26 artifact.
07 — Next stepCompare experiments that support the same question.
Compare experiments that support the same question.
When a benchmark changes, preserve the old evidence and explain the action required for each affected result. An unresolved comparison is more useful than a precise-looking ranking built from incompatible runs.