Hide the model names, give reviewers the same practical task and ask what happened. Record whether they could use the design before asking which version they preferred. Reveal the generator only after those judgments are saved.
This is a proposed comparison method for agent-generated interfaces, reviewed September 12, 2026. No blind study was conducted for the article. The aim is a better local buying decision, not a new model leaderboard.
- 01Hide identity until after judgment.Keep an independent mapping from artifact labels to model and run.
- 02Separate function from taste.An attractive interface can fail the task or exclude users.
- 03Compare workflows fairly.Record the brief, budget and revisions that produced each candidate.
- 04Freeze before you reveal.Save the reviewer record, then bring in model names, pricing and generation time; the order is what keeps the judgment about the design.
01 — Practical decisionWhy hide the model name at all?
The Chatbot Arena paper describes anonymous, randomized pairwise comparisons with identities revealed after voting. That is evidence of a preference-evaluation design for conversational systems. It does not establish that the same procedure has been validated for choosing business interfaces.
Our proposed adaptation is narrower: reduce the information about the generator that reviewers receive before judging an artifact. A famous model, a costly tier or a favored vendor can otherwise become part of the argument before anyone tries the interface. Hiding the label does not remove every bias, and visual habits may still suggest a source.
The comparison unit should be a delivered design with a defined task. Do not ask reviewers to choose the best AI model in general from two pages. Ask which candidate better supports the task you actually need, and what evidence supports that choice.
02 — MethodFix the brief and identify the actual treatment
Give every candidate the same audience, content, brand constraints and user journey. Supply the same permitted assets. Specify what must work, what may be simulated and what must remain editable. The design-file versus screenshot guide helps define those inputs.
Choose whether you are comparing equal spending, equal elapsed time or each workflow’s normal production process. None is automatically fair for every question. Equal prompts with radically different revision budgets do not isolate a model difference. Normal production workflows may deliberately include different tools, but that needs to be the declared comparison.
Preserve the generated source and every revision selected for review. Avoid presenting one candidate’s first draft against another candidate’s carefully repaired final version without saying so. The table below separates three judgments that should remain visible throughout the decision.
| Judgment | Question | Evidence to retain |
|---|---|---|
| Task success | Can the person complete the intended action? | Observed result, blockers and assistance. |
| Accessibility | Can relevant users operate and understand it? | Human checks plus scoped tool findings. |
| Visual preference | Which presentation better serves the brief? | Recorded preference with concrete reasons. |
| Delivery cost | What did accepted output require? | Attempts, repair time, charge and editable files. |
03 — The evidencePrepare anonymous artifacts that remain usable
Assign neutral labels to artifact versions and store the model/run mapping away from reviewers. Remove visible model branding and obvious generator labels where doing so is lawful and does not alter the product being tested. Keep required credits and licensing information intact.
Vary presentation order across reviewers and record it. A reviewer seeing one candidate first may use it to learn the task, making the second easier. For interactive work, use equivalent starting state, data and browser conditions. Do not share login state or a completed form between candidates.
If a reviewer recognizes a likely model from its style, let them record that suspicion without confirming it. Complete blinding may be impossible. The useful record states what was hidden, what could still identify the generator and whether the judgment was committed before reveal.
04 — The decisionAsk people to complete a task before scoring appearance
Give a concrete task: find a service price, compare two options or recover from a failed form submission. Record the observed outcome and where the reviewer needed help. These are proposed task examples, not results. Avoid invented stopwatch figures or a single overall score with unexplained weights.
Check keyboard use, labels, focus visibility, readable text and relevant responsive states. W3C’s accessibility overview states that no tool alone determines whether a site meets accessibility standards; knowledgeable human evaluation is needed. A blind preference vote is likewise not an accessibility certification.
After task checks, ask about hierarchy, clarity and visual preference. Require a short explanation tied to the artifact. A comment that the pricing distinction is difficult to find is more actionable than saying a candidate feels less premium. Keep severe functional failures visible even if the same design wins the taste vote.
05 — ExampleMake ties and disagreement useful evidence
Allow a tie and an insufficient-evidence response. Forcing a choice can create a winner that nobody meaningfully prefers. If reviewers disagree, inspect whether they performed different tasks, used different devices or valued different parts of the brief. Do not average away a failure affecting a critical audience.
An illustrative result could favor one candidate for navigation and another for editing existing content. That suggests a revision or a workflow choice, not necessarily a single winner. Keep the evidence by task so the final decision can reflect the business’s actual priority.
The editable-export reference covers what happens after selection. A design that reviewers like may still be costly to maintain if the team cannot edit its source. Treat that as a separate delivery criterion and disclose it before making the purchase decision.
06 — The costReveal the labels and compare the cost of accepted delivery
Freeze the reviewer record before revealing model names, pricing or generation time. Then bring in the operational evidence: attempts, cost, repair effort, editable files and any required dependencies. This sequence keeps those facts available for procurement without allowing them to substitute for direct design judgment.
A small internal comparison supports a decision about the tested brief and workflow. It does not establish statistical superiority across products or customers. If the difference is slight, prefer a reversible choice or gather more representative cases rather than turning a fragile preference into a public claim.
Use the interactive-demo guide to make the review artifact executable. In an AI transformation project, the deliverable should include both the chosen design and the evidence explaining why it fits the job.
As-of date. September 12, 2026: sources retrieved and reviewed. The article is dated September 12, 2026. Verified event dates are stated separately.
Sources and method. Original decision procedure informed by the Chatbot Arena paper and W3C accessibility evaluation guidance. Sources checked September 12, 2026.
Limits. No reviewers recruited, candidates generated or scores collected. Model anonymity reduces one influence; it does not prove absence of bias.
07 — The failure caseFour blind tests that were not blind
Hiding the label is the easy part; keeping the comparison fair is where blind reviews fail. The illustrative tests below each kept the names hidden and still compared the wrong things. They are design examples, like the rest of this guide, not results of a study.
The draft against the final
One candidate’s first output is placed beside another’s third carefully repaired revision, and reviewers prefer the second. The comparison basis was never declared, so the test measured revision budget, not design.
The forced winner
Reviewers must pick one. Three of five choose candidate A by a hair and nobody meaningfully prefers it. A tie and an insufficient-evidence option would have shown that the decision needed more cases, not a coin flip.
The taste vote over the failed task
Candidate B wins the preference vote and a reviewer could not find the pricing distinction on it. The severe functional failure is averaged away by the appearance score. Task success had to stay a separate, visible judgment.
The early reveal
A procurement note with each candidate’s cost and generation time is circulated before review. The cheaper candidate wins. The operational evidence was meant to arrive after the frozen record, not to substitute for design judgment.
Each test hid the model name and leaked something else. The protocol in the next section lists the order of operations that keeps the leaks out.
08 — The methodA one-page blind review protocol
The guide’s sections reduce to a protocol a team can follow without re-reading it. Fix the brief: same audience, content, brand constraints, journey and permitted assets for every candidate, with what must work, what may be simulated and what must stay editable written down. Declare the comparison basis: equal spending, equal elapsed time or each workflow’s normal production process, and preserve the generated source and every revision selected for review. Anonymise: neutral labels, the model-and-run mapping stored away from reviewers, visible branding removed where lawful, required credits kept. Randomise presentation order per reviewer and record it, with equivalent starting state, data and browser conditions, and no shared login or completed form between candidates.
Then review in order. Task first: a concrete task per reviewer, with the observed outcome and where help was needed recorded. Accessibility second: keyboard use, labels, focus visibility, readable text and responsive states, by a knowledgeable person with scoped tool findings as support, since W3C says no tool alone can determine whether a site meets accessibility standards. Preference third: hierarchy, clarity and visual preference, each with a short explanation tied to the artifact. Allow a tie and an insufficient-evidence response throughout, and let a reviewer record a suspected model without confirming it. Freeze the record. Only then reveal the labels and bring in attempts, cost, repair effort, editable files and dependencies as the delivery-cost judgment.
An illustrative run for two landing-page candidates: brief and basis fixed on Monday, four reviewers each given the same two tasks in randomised order, records frozen before the Friday reveal, and a decision that favours one candidate for the pricing task and the other for editing existing content. That is a revision or a workflow choice rather than a winner, as the ties section says, and the protocol is what made the difference visible.
It removes the model name, price and vendor from the argument before anyone tries the interface. It does not remove order effects, prior experience or a recognisable style, so the record states what was hidden, what could still identify the generator, and that the judgment was committed before the reveal.
09 — Next stepHiding the model name shows what the design is worth
Run the protocol once on a real brief with the candidates and reviewers you have. The record it produces will decide the brief in front of you, and the parts of the protocol that felt unnecessary are the ones the four failures above depend on.
Choose the design on evidence, then choose the workflow
Anonymous review can make the artifact the focus of attention. Keep task success, accessibility, preference and delivery cost visible as separate judgments. Reveal the generator after review, then choose the workflow that produces an acceptable result under your real constraints.