Cost-Per-Successful-Task: A new AI Evaluation Metric

$/token has been the dominant AI procurement metric since the GPT-4 era. It is also wrong. A model that is 19× cheaper per token but needs 3× as many retries to hit the same correctness bar may be more expensive in production than the headline rate suggests, less expensive than the headline suggests, or roughly the same — and you cannot tell from the rate alone.

Cost-per-successful-task (CPST) is the right unit. It is the total dollar cost to complete a real task end-to-end — input tokens, output tokens, tool-call cost, retry cost, cache write amortization — divided by the count of tasks actually delivered to spec. CPST is what production budgets pay and what procurement RFPs should cite.

This piece defines the metric, ships the formula, walks six worked examples across common workflow types, and provides a scoring template ready to drop into your procurement playbook. The metric is simple. The implications for model selection are often the opposite of what $/token suggests.

$/token compares input rate or output rate per million tokens. It is a clean number with a clean unit. It also ignores everything that determines what you actually pay in production:

• Pass rate. A 51% pass-rate model needs roughly twice as many attempts as an 80% pass-rate model on the same workflow. The cheaper-per-token model is often more expensive in total.
• Retry inflation. Failed attempts are not free in production — they cost full input + output every time. Even if retries succeed at 95% on attempt 2, the cost is 1.4-2.8× the first-attempt cost.
• Output amplification. Long-context workflows elicit longer outputs. A model that is cheaper per output token but produces 3× more output per task may be more expensive net.
• Tool-call overhead. Every tool call adds context tokens for the tool spec, the call itself, and the response. On chained workflows, tool-call overhead can be 30-60% of total spend.
• Cache and batch tier dynamics. Effective rate after cache hits and batch usage often differs from rack rate by 60-95%. Procurement that compares rack rates is comparing numbers neither team will pay.

CPST measures the dollar cost paid to deliver one task end-to-end to your defined quality bar. Three components:

Stripped to a single line:

CPST = Σ(input_cost + output_cost + tool_cost + retry_cost + cache_amortization) / pass_count

Where:

• input_cost = input tokens × input rate, summed across all attempts (failed + successful).
• output_cost = output tokens × output rate, summed across all attempts.
• tool_cost = sum of tool-call input/output token costs, including tool spec context overhead.
• retry_cost = redundant input tokens on retried calls, often dominated by re-sending the original prompt.
• cache_amortization = cache write cost spread over the cache reads it enables (write × 1 / read_count).
• pass_count = count of attempts (or batches) that passed the quality bar within the retry budget.

Six common AI workflows, each with the CPST math worked out across model choices.

Three telemetry primitives are required to compute CPST in production.

Most APM platforms (Datadog, Grafana, Honeycomb) now ship LLM telemetry primitives. The cost telemetry is solved — provider response headers carry token counts. The success-label instrumentation is the gap. Build it as a first-class workflow primitive: every workflow returns a success_label alongside its output. Without it, your AI ops dashboard reports cost-per-call, not cost-per-success.

For RFPs and vendor evaluations, the scoring template below converts CPST into a comparable procurement metric across model+ provider options.

Six failure modes we see when teams first adopt CPST.

• Skipping success classification.Reporting cost-per-call as CPST. The metric is meaningless without a success denominator. Either build the verification logic or don't use the term.
• Setting the quality bar at evaluation time, not production. Picking a bar that's easier to hit on the eval set than on production traffic. CPST then looks great in eval and bad in production.
• Ignoring retry math. Computing CPST off successful attempts only, ignoring the cost of failed attempts that preceded them. Inflates the apparent advantage of low-pass -rate models.
• Comparing across different quality bars. Two models scored against different definitions of success. CPST comparisons require a single quality bar; otherwise the numbers are not commensurable.
• Not segmenting by workflow class. Reporting a single CPST across all production traffic. The number is real but useless for procurement — different models win different classes.
• Treating CPST as the only metric.CPST is cost-bound; latency, security, and capability ceiling are orthogonal SLOs that need separate gates. Combine, don't substitute.