A Cheaper AI Model Can Leave You With More Review Work

A cheaper model can cost more per accepted task if people spend the difference inspecting, correcting and rechecking its work. It can also be the better choice. You need a record of the review work to tell which is happening; the inference bill alone cannot settle the decision, because the inference bill is the one cost that arrives itemized and the review cost is the one that arrives as a tired team.

Keep the acceptance standard fixed and compare the complete route to a usable result. The ledger below is a proposed measurement method, not evidence that inexpensive models are generally worse or that a particular model saves money. This guide sets out what to record, which research supports measuring rather than assuming, how to keep the comparison interpretable, the break-even arithmetic, the ways a ledger misleads, and how to turn the result into a routing decision rather than a verdict on a model tier.

Model pricing is public, per token, and shows up on an invoice. Review cost is private, per interruption, and shows up as a reviewer who got less done this week. When a team compares two models it compares the number it has, and the number it has is the one that favors the cheaper model by construction. The expensive part of a cheap model is the part nobody wrote down.

The fix is not to assume the opposite. A cheaper model that produces usable output with the same review effort is simply cheaper, and penalizing it for its price would be as wrong as ignoring the review it causes. The fix is to make the second bill as visible as the first: record the human time by activity, attach it to the task, and divide the whole by what was actually accepted.

Use one record per assigned task, with any model reruns attached to that same record. The categories below are our operational definitions. Apply them consistently across candidates rather than adjusting the categories to fit a preferred result; the comparison is only as good as the discipline of putting the same kind of minute in the same row for both models.

The right-hand column is what to keep out of each row. Most of the errors in a review ledger are not missing minutes but minutes counted twice or counted in the wrong place: a rerun booked as a correction, a queue booked as inspection, a rejected draft quietly dropped so the average looks better.

Anthropic’s agent-evaluation guide distinguishes code-based, model-based and human grading, with different costs and limitations. That supports naming the kind of review performed rather than treating every check as an interchangeable quality score. A minute spent running a deterministic check and a minute spent reading for nuance are both review, and they belong in different rows because they respond differently to a change of model.

METR’s July 2025 developer study measured work in a specific setting using early-2025 tools. The authors explicitly limit generalization beyond those developers, repositories and tools. We use it as a reason to measure actual work, not as a current estimate of model productivity: its contribution here is methodological, a demonstration that perceived and measured effort can diverge enough to make measurement necessary.

Neither source supports a rule that a cheaper model creates more review. Prompt design, source access, task difficulty and the required output can change the result. The title describes a possibility to investigate, not a finding about a model tier.

Choose representative tasks and write the acceptance rule before the outputs arrive. Keep source access and output requirements comparable. Record whether the reviewer knows which model produced the result; expectations can influence how much scrutiny they apply, and a reviewer who knows which draft came from the cheap model may read it harder, or softer, than its content warrants.

Separate active time from elapsed time. A reviewer might spend a short interval checking a draft that waited until the next morning. The labor cost follows active work; the delivery delay follows elapsed time. Both matter, but adding them together would count waiting as paid effort without justification, and the model with the slower queue would look expensive for a reason that has nothing to do with its output.

Keep difficult tasks and rejected outputs. If one model’s weak drafts are removed before computing the average, the comparison rewards the model for work that did not make it through review. Report task mix and acceptance counts alongside any average, so a reader can see whether a low cost per accepted result came from good drafts or from a generous definition of accepted.

For a defined task set, add inference spending, tool spending and active review cost, then divide by the number of accepted deliverables. Active review cost is review minutes multiplied by the chosen hourly labor rate and divided by sixty. State whether that rate includes overhead. If nothing is accepted, cost per accepted result is undefined, not zero.

A model’s inference saving can be compared with its extra review cost only when the deliverable count and quality standard are comparable. The break-even additional review minutes equal the inference saving multiplied by sixty and divided by the hourly review rate. This is arithmetic, not a forecast; insert your measured inputs and disclose them. An illustrative run of the formula: if the cheaper model saves $3 of inference per task and review is costed at $90 an hour, it can absorb two extra review minutes per task before the saving is gone, and a third minute makes it the more expensive option.

Show a range when labor rates or review timing are uncertain. Do not turn a small trial into a precise annual saving. The API price index supplies a different input; it cannot tell you how much checking your team needs.

A ledger that is kept carelessly produces a confident wrong answer, which is worse than no ledger because it ends the discussion. The illustrative patterns below are the ones the right-hand column of the table is written to prevent.

The survivorship cut: rejected drafts are left out because they “did not count”, so the cheaper model’s average is computed over its best work. The queue-as-effort error: a draft that sat overnight is booked as eight hours of review, and the slower model looks ruinous for a scheduling reason. The double rerun: a model retry is logged as a correction, so the inference spend appears a second time as labor. The informed reviewer: the person checking knows which draft is the cheap one, and the ledger measures their expectation as much as the output.

Each is detectable from the record itself. If the rejection row is empty for a model that had failures, the cut happened. If an inspection interval exceeds a working day, waiting was counted. If correction minutes exist with no human edit attached, a rerun was double-booked. And whether the reviewer was blind is a field the ledger should carry per task.

If a lower-cost model produces usable results with similar review effort, there is no reason to penalize it for being inexpensive. If it creates frequent specialist escalations, the scarce resource may be specialist attention rather than compute spending, and the ledger’s escalation row is the one to read before the total.

Use separate results for distinct task types. A model can be suitable for formatting verified material and unsuitable for source-heavy synthesis. Change the route for the affected work instead of declaring a universal winner; the same ledger that shows a model losing on synthesis can show it winning on reformatting, and both results are the answer.

Use the model-switch testing guide for the broader evaluation setup and the reviewer-independence guide when adding a second automated reviewer. More review steps should earn their cost by answering a defined question, and that cost belongs in the ledger too. Teams that want the ledger and the routing rule built into their agent workflows rather than kept in a spreadsheet can work with our AI transformation practice.