Terminal-Bench 4.0: When Agent Scores Need a New Test

Terminal-Bench 4.0 changes the conditions behind an agent score. Before using an older result to choose a coding agent, identify which tasks and settings changed, then decide whether the evidence requires a new run, a new grade or no substantive change. A model name and a percentage are not enough.

The official v4.0.0 release was published on August 26, 2026. It identifies removed tasks and changes to existing ones. The useful buying question is which of your comparison rows remain supported, rather than whether the new version makes an agent look better.

Treat an existing result as evidence produced by a specific experiment. Rerunning repeats the agent’s work. Regrading applies an appropriate verifier to retained outputs. Reuse carries a result forward when the relevant experiment has not materially changed. Each route needs a reason.

The benchmark maintainers describe these distinctions in their continuous-benchmark methodology . Apply the change classification to the affected tasks. A major dataset update does not by itself establish that every individual task needs the same treatment.

A concrete example appears in the pinned v4.0.0 task for batched-eval-parity. Its README summary describes four hours and one CPU, while the changelog describes eight hours and two CPUs. The task configuration specifies an agent timeout of 28,800 seconds and two CPUs.

That is a documentation/configuration discrepancy observed by reading the pinned files, not a benchmark run. It shows why the source and revision matter. The configuration is more directly relevant to execution than a stale summary, but even it does not prove the settings of a particular submitted result. Overrides may still apply.

Ask for the actual run configuration: resource allocation, hard limits, timeout, concurrency, agent revision and retry policy. Keep the source of every field. If a provider reports its default rather than the run’s value, mark that difference instead of filling the gap with an assumption.

Anthropic’s infrastructure study reports that resource settings changed results in its Terminal-Bench 2.0 experiments while the model, harness and tasks were held fixed. It distinguishes guaranteed resources from hard limits. This is vendor-run evidence about that experiment, not a universal correction factor for version 4.0.

A larger allowance can change what an agent has time to attempt, while a hard limit can determine whether a process survives. Record both when they matter. Do not infer that increasing every resource always improves the score or that one percentage adjustment makes unlike experiments comparable.

Our OSWorld and Terminal-Bench explainer covers what the benchmark families measure. This article adds the migration question: how to preserve a defensible comparison when the measurement system itself changes.

Give each old-to-new comparison a record with its own inclusion rules. If the new dataset removes tasks, a shared model label does not preserve the old denominator. A score is meaningful only with the task set and outcome accounting that produced it.

Suppose a supplier shows an older result beside a version-4 score from another agent. Ask for the common task set, each run’s configuration and the policy used to carry results forward. If those are unavailable, treat the comparison as incomplete. You can still investigate both products without pretending their numbers are directly comparable.

The vendor benchmark reproducibility audit provides a disclosure baseline. The broader SWE-Bench and Terminal-Bench guide explains their differing tasks. Neither source should be used to manufacture a conversion between benchmark versions.

Avoid importing current leaderboard percentages into a historical comparison without a dated export. A present-day board can change after a submission correction or new run. Retain the original record that supported the decision, including unsuccessful attempts and missing fields.

Start with the result you actually use to shortlist an agent. Classify its relevant changes and request only the missing evidence. If a new run is needed, freeze its task set and settings before comparing it with another system. If regrading is possible, retain the old artifacts and both verifier outputs.

No benchmark tasks were executed for this article, and no model ranking is inferred from the release. The migration table is a proposed editorial method. For help connecting evaluation evidence to a working application, our AI transformation service can scope an evaluation around the tasks that matter to your team.

As-of date. September 12, 2026. The article is dated September 9, 2026; this research was reviewed later.

Method. Primary release documentation and pinned task files were inspected. The result-migration table is a proposed method for buyers; it is not a live leaderboard.

Limitations. No benchmark runs, model comparisons or cost measurements were performed. Current methodology pages provide context; release-specific claims use the pinned August 26 artifact.

A benchmark version change breaks comparisons quietly, because the model name and the percentage survive it. The illustrative comparisons below each carry a number across the line without the evidence that would let it travel. They are patterns the migration table anticipates, not vendor claims we have seen.

Each comparison looked precise and rested on incompatible experiments, which is the conclusion’s warning. The record in the next section is the shape of a comparison that can be audited.

The fourth section asks for a record per old-to-new comparison, and the sixth asks for the migration to stay narrow enough to audit. Together they define the record for the one result you actually use to shortlist an agent. Dataset identity: the old and target versions with their immutable references and task manifests, so additions, removals and changed tasks are visible. Agent identity: harness revision, model identifier, reasoning and retry settings. Execution: the resources and hard limits as run, with the source of each field and the gaps marked where a provider reported a default rather than the run’s value. Change classification: for each affected task, whether the input or environment changed, the verifier changed or nothing material changed, with the action that follows, rerun, regrade or reuse. Retained artifacts: what exists to regrade from and which verifier versions produced which outputs. And a dated export of any leaderboard figure cited, kept with the record rather than refreshed.

An illustrative record for one shortlisted result reads: dataset, the 2.0 manifest against the pinned 4.0 release of August 26 with the removed and changed tasks listed; agent, the harness revision and model identifier with reasoning settings as the provider reported them and the retry policy marked unknown; execution, the resource allocation and timeout from the task configuration at the pinned revision, with the README discrepancy noted; classification, eleven tasks unchanged and reused, four with environment changes rerun under frozen settings, two removed and dropped from the denominator; artifacts, final outputs retained for the rerun set; board export, dated the day the shortlist was made.

The record does not produce a single percentage, and that is its value. It produces a comparison whose unresolved parts are visible, which the conclusion says is more useful than a precise-looking ranking built from incompatible runs.

Pick the one benchmark result your shortlist actually depends on and write its record before the next vendor meeting. The fields that cannot be filled are the questions to ask, and the answers decide whether the number crosses the version line at all.