Give AI Reviewers Different Checks Before Trusting Them

Give AI reviewers different jobs before trusting their approvals. Have one recompute a calculation while another checks its denominator and units; have one reproduce a user action while another inspects the saved result. Two reviewers asked the same question of the same draft can make the same mistake, and two matching approvals then look like confirmation when they may be one opinion counted twice.

For the research on shared errors and judge bias, see our cross-model review method. This companion focuses on assigning the checks and recording what each one establishes: which risk each reviewer is responsible for, what inputs it needs to do that job honestly, how to keep the first verdict from steering the second, and what to do when the two disagree. The assignment table below helps a builder or editor choose checks that expose different mistakes.

Reviewers share errors when they share inputs, framing and shortcuts. Two models reading the same draft receive the same misleading sentence, the same confident number and the same ambiguity in the brief, and each may resolve them in the most plausible way, which is often the same way. Adding a reviewer from a different vendor changes the model and leaves everything else in place. The approvals agree because the mistake was upstream of both of them.

The remedy is not more reviewers but different ones: different inputs, different tasks, different evidence. A reviewer that recomputes a figure from its source cannot inherit the draft’s arithmetic error because it never reads the draft’s arithmetic. A reviewer that opens the saved record cannot inherit the narrated success because it is looking at the state, not the story. Each additional check is worth having only if it can fail where the previous one could not.

Begin with the claim or behavior whose failure would change the decision. Then give each reviewer an evidence-producing assignment. The table is a proposed review design, not a measured comparison of model combinations. Each row names a way a draft can be wrong, the check that catches the obvious version of it, and the complementary check that catches the version the first one lets through.

Notice that the complementary column rarely asks “is the first review right?” It asks a different question about the same risk. For a calculation the first check is the arithmetic; the complementary check is whether the arithmetic answers the question the article posed. Both can be done by a model, and neither can be done by reading the other’s verdict.

Consider an illustrative draft: a metric rises from 40 to 50, so the draft calls it a 10% increase. A reviewer that reads only the prose may approve it; the sentence is grammatical, the numbers are in the right order and ten is indeed the difference. A reviewer that starts from the inputs can distinguish an absolute increase of 10 units from a relative increase of 25%: (50 − 40) ÷ 40 × 100.

If the metric itself is a percentage, such as a rate moving from 40% to 50%, the absolute movement is 10 percentage points. That is a third expression with a different meaning. The correct label depends on the question the article is answering, which is why the complementary check in the table asks about the denominator rather than repeating the arithmetic: the arithmetic can be right and the label still wrong.

Do not ask the second reviewer merely whether the first reviewer is reasonable. Give it the inputs, the intended claim and the acceptance rule. The citation verification reference provides a similar separation for source-backed prose: one check finds the passage, the other asks whether the passage says what the draft says it says.

A reviewer that is shown “reviewer one approved” before it starts has been given a strong prior and a social reason not to contradict it. The same risk applies to model reviewers: a second review that can see the first verdict may simply confirm it, with reasoning that paraphrases the first. The fix is procedural rather than clever. Run the two checks without sharing verdicts, then compare.

An illustrative sequence. Reviewer one is given the draft and the source and asked to find the supporting passage for each claim. In parallel, reviewer two is given the source and the list of claims with the draft’s wording removed, and asked to state what the source supports about each topic. Neither sees the other. A reconciliation step then lays the two outputs side by side; the claims where they differ are the ones that need a person.

Reconciliation is where the verdicts finally meet, and it should compare evidence, not conclusions. “Reviewer one says supported, reviewer two says the source covers a narrower population” is a useful disagreement because it names the passage and the gap. “Reviewer one says yes, reviewer two says no” is not.

When findings differ, compare the input revision and acceptance criterion before requesting another review. Some disagreements turn out to be two reviewers looking at different versions of the draft, or reading an acceptance rule that each interpreted differently because nobody wrote it down. Those are fixed by alignment, not by a third opinion.

For the disagreements that survive, record the specific unresolved question and the evidence or owner decision needed to settle it. A question of fact goes back to the source. A question of interpretation goes to whoever owns the claim. A question of acceptance goes to whoever owns the deliverable. Routing the question is the review’s output; another vote would only hide that the question exists.

For behavior claims, a replayable check is especially useful. The model trial guide explains how task-specific evidence improves a decision beyond a public ranking, and the same replayable case settles a review disagreement faster than a debate about it.

Each of these illustrative patterns produces a second approval and no new evidence, and each is easy to fall into because it is the path of least configuration.

Same prompt, different model: the two reviewers were given identical instructions and the draft, so they share every ambiguity in the instructions. Verdict in the context: the second reviewer’s prompt includes the first’s approval, and the second review becomes a search for reasons to agree. Reviewing the review: the second reviewer is asked whether the first review is reasonable, which it is, because reasonableness was never the question. Summary instead of inputs: the second reviewer is given the agent’s change summary rather than the source or the saved state, so it inherits the summary’s framing and cannot see what the summary left out.

An illustrative case of the last one: a draft’s change summary says “updated the Q3 figure to the revised value”. Both reviewers approve the update. Neither notices that the chart still plots the old figure, because neither opened the chart; they reviewed the sentence about the change. A reviewer assigned to open the artifact, with no summary at all, would have found it quickly.

Anthropic’s evaluation guide distinguishes code-based, model-based and human graders. Use a direct calculation for a total, a model for a nuanced interpretation and the responsible person to settle an ambiguous requirement. The cheapest grader that can produce decisive evidence is the right one; a model asked to re-add a column is an expensive calculator, and a calculator asked to judge tone is a broken one.

Record the claim checked, evidence inspected, finding and unresolved limitation. Reuse that record when the draft changes so a later editor can tell which checks remain valid; a correction to one number invalidates the arithmetic check and not the source check, and the record is what tells you which. For the artifact itself, use the file acceptance reference. The same assignment logic applies when the thing under review is a factual correction rather than a draft; our guide to changing facts without losing context shows where the second check belongs. Teams that want review assignments designed into their agent workflows rather than improvised per draft can work with our AI transformation practice.