Agent Success Rate (ASR)

Agent evaluation is in a strange place in 2026. The benchmark suites — τ-bench, Berkeley Function Calling, SWE-bench Verified, MLE-bench — give us model-level numbers. Production teams running agents in real workflows are stuck between three options: roll their own metric (different per team, not comparable), report a simplistic pass-rate (treats a partially correct answer as a failure), or use a cost-blind metric (treats a $4 task that succeeded as equivalent to a $0.04 task that succeeded).

Agent Success Rate (ASR) is the methodology we ship to clients to close those gaps. It is not a benchmark suite — it is a measurement spec that runs on top of any benchmark suite or production trace store. The spec defines the outcome classes, the partial-credit weighting, the cost-penalty function, the confidence-interval reporting, and the reference dataset of 200 production-agent traces we use as a calibration set.

By Q1 2026, most production agentic teams report some flavour of success rate. The metrics are not comparable. Some teams use strict pass-rate (binary, no partial credit). Some teams use task completion (allows partial). Some teams report cost per successful task without any partial credit. Some teams report a qualitative reviewer score on a 1-5 scale. The result: when a VP-level conversation needs to compare two agent versions or two providers, no apples-to-apples answer exists.

ASR is the methodology that closes the gap. It does not invent a new benchmark; it specifies how to compute a comparable rate from any benchmark or production trace store, with partial credit and cost-adjustment baked in.

The full ASR formula:

ASR = (n_full + 0.4·n_partial+ − Σ cost_penalty_i) / n_total

  cost_penalty_i = max(0, (cost_i − budget_ceiling) / budget_ceiling)
                   capped at 1.0 per task

  outcome ∈ {complete, partial+, partial−, hallucinated, abandoned}

ASR is a rate in the range [0, 1], reported as a percentage. Higher is better. A perfect agent on a basket of in-budget tasks scores 100%. A partially-effective agent on a basket of slightly-over-budget tasks might score 38%. The formula degrades gracefully and does not produce negative values.

Reviewers label each agent run into exactly one of five classes. The labels are structured to make reviewer agreement high (binary decision tree at each level) and to preserve diagnostic signal for downstream analysis.

The cost-penalty function is the part of ASR that practitioners push back on the most. The pushback is fair — penalising an agent for being expensive can produce metrics that prefer cheap-and-bad over expensive-and-correct. The design below threads that needle: costs only penalise above a per-task budget ceiling, and the penalty is linear-up-to-1.0, never producing negative scores.

The reference dataset is six task families × 30-35 traces each. Its purpose is reviewer calibration, not benchmarking. Teams use it to align internal reviewers on what counts as partial-vs-incorrect-vs-hallucinated; once calibrated, they apply the same labels to their own production traces.

ASR without confidence intervals is noise. The default reporting format includes the headline number, a 95% bootstrap CI, the outcome-class breakdown, and the cost-variance side panel. The template below is what we ship to engineering leads weekly.