Two AI reviewers agreeing on an answer does not establish that either checked the fact that matters. If both read the same misleading summary, both can repeat its mistake. Add a second reviewer when it brings a different check: opening the original source, recalculating the number or testing the claimed behavior.
Correlated errors are mistakes that occur together because the reviewers share an input, assumption or failure mode. You do not need a statistical estimate of correlation to notice the practical problem: two approvals based on one unchecked claim still leave that claim unchecked.
- 01Assign different checks.Independent calculations and source inspection add evidence that another opinion may not.
- 02Keep the first verdict hidden when useful.A reviewer should form its own finding before inheriting the prior conclusion.
- 03Resolve disagreements with evidence.Another vote cannot repair a missing source or an ambiguous acceptance rule.
01 — Replace two approvals with two useful jobsReplace two approvals with two useful jobs
Begin with the claim or behavior whose failure would change the decision. Then give each reviewer an evidence-producing assignment. The table is a proposed review design, not a measured comparison of model combinations.
| Risk in the draft | First review | Complementary review |
|---|---|---|
| Wrong calculation | Recompute from the original inputs | Check whether the denominator and units answer the question |
| Unsupported source claim | Find the exact supporting passage | Compare the passage’s population and period with the draft |
| Broken user action | Execute the acceptance case | Inspect the resulting saved state |
| Misleading comparison | Check each factual entry | Check whether the criteria favor one option without justification |
| Lost qualification | Compare the draft to the source | Read the conclusion without the supporting section |
| Unusable export | Inspect content completeness | Open the recipient’s final artifact |
02 — What judge research can and cannot tell youWhat judge research can and cannot tell you
Zheng and colleagues’ 2023 LLM-as-a-judge paper examines position, verbosity and self-enhancement biases, alongside reasoning limitations. Its experiments concern particular models and evaluation settings. We do not treat historical agreement with human preferences as a reliability guarantee for today’s business reviews.
Anthropic’s agent-evaluation guide separates code-based, model-based and human graders. Our recommendation is to select checks for the property being evaluated. A model can assess a nuanced explanation while a calculation is recomputed directly.
This does not mean AI review is pointless. It means the review result should name the evidence and the criterion it checked. “Looks good” is weaker than “the cited table contains this value for this period.”
03 — Give the second reviewer the original inputsGive the second reviewer the original inputs
Consider an illustrative draft: a metric rises from 40 to 50, so the draft calls it a 10% increase. A reviewer that reads only the prose may approve it. A reviewer that starts from the inputs can distinguish an absolute increase of 10 units from a relative increase of 25%: (50 − 40) ÷ 40 × 100.
If the metric itself is a percentage, such as a rate moving from 40% to 50%, the absolute movement is 10 percentage points. That is a third expression with a different meaning. The correct label depends on the question the article is answering.
Do not ask the second reviewer merely whether the first reviewer is reasonable. Give it the inputs, the intended claim and the acceptance rule. The citation verification reference provides a similar separation for source-backed prose.
04 — Turn disagreement into a specific unresolved questionTurn disagreement into a specific unresolved question
When reviewers differ, ask what observable fact would settle the issue. One may have used the wrong revision; another may have applied a stricter criterion. Preserve both findings until the evidence or the owner’s requirement resolves that difference.
If the source is inaccessible, another model’s recollection is not a replacement. Mark the consequential claim unverified and narrow the conclusion. If the criterion is subjective, ask the person responsible for the output to choose the standard rather than pretending there is one objective answer.
For behavior claims, a replayable check is especially useful. The model trial guide explains how task-specific evidence improves a decision beyond a public ranking.
05 — Spend review effort where it changes acceptanceSpend review effort where it changes acceptance
A second opinion is useful for ambiguous interpretation, competing alternatives or missed requirements. A deterministic check is often more direct for a total, a broken link or a file that will not open. Start with the failure you are trying to catch and select the reviewer afterward.
Record the claim checked, evidence inspected, finding and unresolved limitation. Reuse that record when the draft changes so a later editor can tell which checks remain valid. For the artifact itself, use the file acceptance reference.
06 — DecisionWhat to do next
Ask what the second review adds.
Keep a second reviewer when it contributes a distinct check or perspective. Acceptance should rest on the evidence those checks produce, with unresolved judgments left visible.
For implementation support, explore our AI transformation services.