Did AI Double Research Speed? What METR Actually Found

METR’s November 5 assessment offers evidence that AI development at Anthropic is accelerating. It does not establish that an ordinary business can halve delivery time by adding an AI assistant. The useful response is to take the finding seriously while asking which measure changed, what explains it and whether the same mechanism exists in your work. METR’s assessment

For a founder deciding where to invest, this is more useful than either extreme: treating a striking research result as a guaranteed productivity multiplier, or dismissing it because the evidence is incomplete. A good external result should improve the questions you ask before spending money on a local programme.

METR assessed Anthropic from August 12 to October 29. Its report contains several kinds of evidence, which should not be combined into a single productivity percentage. AECI is Anthropic's version of the Epoch Capabilities Index; the reported trend compares benchmark growth since October 2025 with the earlier rate.

METR says Anthropic redacted most of the evidence it considered especially informative. The report does not establish independently measured, end-to-end productivity gains for ordinary businesses. Assessment and limitations

The table’s distinctions are the point. A statistical trend can be estimated from observations. An explanation of that trend requires a causal argument. A broader judgement can draw on several incomplete sources. Each may be useful, but they answer different questions and deserve different confidence.

When presenting a result to a leadership team, retain the measure in the sentence. “The rate of improvement on this capability index increased” tells the reader what changed. “AI doubled productivity” removes the denominator, the setting and the uncertainty. That shortening makes the statement easier to repeat while making it less useful for a decision.

A capability index aggregates performance on tasks chosen by its designers. A business delivery process includes other activities: choosing a problem, collecting context, producing work, reviewing it, resolving disagreements and putting the result into use. Progress on one measure can be highly relevant without being identical to progress through that entire process.

Imagine a team whose limiting problem is solving difficult implementation tasks. A stronger model might unlock work that was previously impractical. Now imagine a team whose work waits for an external decision. The same improvement in model capability may have little effect on the waiting time. These are proposed mechanisms to examine, not findings about the organisations in METR’s report.

The distinction also changes the meaning of “twice as fast.” It could refer to more tasks completed per hour, shorter elapsed time for a fixed task, a steeper benchmark trend or greater capability after a fixed period. Those quantities are not interchangeable.

Write the business question using a specific outcome. For example: how long does it take to produce and accept a working integration that meets a fixed specification? Or: how much reviewer effort is required for a research brief whose factual claims pass checking? These questions preserve the quality condition that makes the work valuable.

An evaluator can exercise independent judgement while some underlying evidence remains unavailable to readers. That does not automatically invalidate the assessment. It changes how much of the conclusion the public can independently reconstruct.

METR’s published approach to working with AI companies separates process transparency, independence and transparency about evidence. It calls for disclosure of scope, access limitations, important redactions and other constraints. This is a useful reading framework because a recognised evaluator’s name does not, by itself, answer those questions.

For a practical evidence review, ask four things. What could the evaluator observe? Which observations are published? What assumptions connect those observations to the conclusion? What would change the conclusion? A report can be candid and valuable while leaving some of those questions unresolved.

Avoid filling a redaction with the explanation you prefer. Missing detail is not proof of either extraordinary progress or a weak result. It is missing detail. The defensible response is to retain the limitation when using the conclusion, and avoid giving a precise local forecast more confidence than its inputs support.

The same standard applies to a vendor’s private customer case study. A compelling claim becomes more useful when the buyer can see the task definition, comparison period, exclusions and acceptance criteria. Confidentiality may prevent full disclosure, but it does not turn an unverifiable claim into a measured result for the buyer’s own setting.

Before commissioning an AI programme, identify what the proposed system is expected to improve. “Make the team more productive” is too broad to guide an evaluation. Useful mechanisms are more concrete: reduce repeated information retrieval, shorten a first implementation pass, catch an error earlier or produce more candidate approaches within a fixed time budget.

Then identify the constraint that could prevent the improvement reaching the final outcome. Faster drafting may create more review work. More candidate ideas may require more selection time. Better code generation may still wait for a decision about what the product should do.

Map one representative job from request to acceptance. Record the steps where someone actively works and the periods when the job waits. This need not become a surveillance exercise. The purpose is to understand the workflow, including handoffs and dependencies, rather than assign every minute to an individual.

Ask the people who receive the output what makes it usable. A developer may value a clear reproducible problem more than a long list of speculative suggestions. An editor may value accurate sources more than a quick first draft. Their acceptance criteria should shape the trial because they bear the cost of incomplete work.

Consider a hypothetical job requiring 20 hours of active work: eight hours to produce an initial artifact and 12 hours for review, correction and integration. If AI halves the first stage while the rest stays unchanged, the total becomes 16 hours. That is a 20% reduction in active work, not 50%.

The example contains invented inputs chosen to illustrate arithmetic. It is not a measurement of a Digital Applied project or an estimate of Anthropic’s work. Its value is that anyone can replace the inputs with their own observed quantities.

Calculation: (20 − 16) / 20 = 20% less active work. If accepted output remains unchanged, the corresponding output-per-hour ratio is 20 / 16 = 1.25. “20% less time” and “25% more output per hour” describe the same simplified scenario with different denominators.

Now change the assumption: suppose review grows because the faster first stage creates more errors or more material to inspect. The gain shrinks. If review becomes easier because the artifact includes better evidence, the gain may grow. Neither effect should be assumed without observation.

Keep elapsed time separate. A job with 16 active hours can still take several calendar days if it waits for approval. Reporting active-work savings as an equal reduction in delivery time would misdescribe the customer’s experience.

This small worksheet is often enough to improve a proposal. It forces the team to specify which stage changes and what remains fixed. It also reveals when an impressive component-level speedup has little room to influence the overall result.

Choose a recurring task with a clear definition of completion. Retain the ordinary workflow as the comparison, and decide how results will be reviewed before running the trial. Avoid selecting only examples that make the new system look capable.

Where practical, assign comparable tasks to the ordinary and assisted workflows. Preserve differences that cannot be controlled and report them. A team may not be able to run a formal randomised study, but it can still avoid comparing a difficult historical project with an unusually easy new one.

Record the following outcomes at the task level:

Review outputs against the same standard. If the new workflow changes what counts as acceptable, describe that change explicitly rather than calling the difference pure acceleration. Likewise, record when AI enables a new class of work that the baseline could not perform. That may be a valuable capability gain even when a simple time-saving comparison is inappropriate.

Keep failed attempts in the denominator. A method that finishes a few selected tasks quickly but leaves others unresolved may not improve the overall service. The cost-per-successful-task guide offers a useful way to connect this acceptance standard with spending.

Early trials often involve changing instructions, learning tools and discovering that the task was underspecified. Those activities are part of adoption, but they should not be confused with the steady operating cost of a mature workflow.

Keep setup effort in the record and show it separately from repeated task execution. A workflow may be worthwhile after enough use even if its first run is expensive. Another may look efficient during a carefully supervised demonstration and become costly when ordinary users encounter edge cases.

Preserve a small regression set: representative tasks that can be rerun when the model, tools or instructions change. A positive result describes a particular configuration. It should not become a permanent claim about every future version of the product.

Look for transferred work. If a writer produces drafts faster but an editor spends more time checking them, the improvement must be assessed across the combined workflow. The same applies when a coding assistant saves implementation time but increases the burden on reviewers or support staff.

Give the pilot a decision date and a concrete next step. The outcome might be expansion, a narrower use case, another controlled iteration or stopping the experiment. Avoid leaving an inconclusive pilot running indefinitely simply because the initial research headline was exciting.

A good investment proposal states both the expected benefit and the evidence that would make the team revise it. For an assisted research workflow, that could be accepted briefs completed with less combined drafting and checking effort. For a development workflow, it could be reviewed changes that meet the same specification with lower total effort.

Use a modest initial commitment when uncertainty is high and learning is cheap. Expand when the trial shows a repeatable benefit on work the business actually needs. Do not force every experiment to justify an organisation-wide transformation.

Keep the conclusion specific. A result on one recurring task can support that task’s workflow. It does not establish a universal multiplier for every employee. Conversely, a failed trial in one setting does not disprove progress elsewhere. The useful unit is the combination of task, system and acceptance standard that was evaluated.