A Successful AI Demo: What Evidence Should a Buyer Ask For?

A successful AI demo is a reason to investigate, not enough evidence to buy. Ask the supplier to run tasks you choose, show any human intervention and deliver the result in the form your team would actually use. Agree on acceptance before the trial starts, while neither side knows which examples will look best, because that is when the acceptance rule is most likely to be neutral.

The buying question is specific: can this agent complete the work you intend to delegate under conditions you can support? A polished presentation can answer a different question, whether a carefully prepared example is possible, and the two answers are easy to confuse when the presentation is good. This guide sets out the evidence packet to ask for, how to run repeated trials without inventing a reliability number, how to count the work hidden around the demo, and how to write a decision that still makes sense after the salesperson has left.

A demo is typically a supplier’s best case, run on inputs the supplier chose, with a narrator who knows where the rough edges are and can steer around them. None of that is dishonest. It is what a demo is for: to show that the thing is possible. The buyer’s mistake is to read “possible” as “reliable on our work”, which is a claim a demo is not designed to support, however many times it is repeated.

The question a buyer actually needs answered has four parts. Can the agent complete our tasks, not the supplier’s? Under our conditions, with our access, our data quality and our review capacity? Without help we will not have after purchase? And with failures we can recognize and absorb? Each part is testable in a pilot and invisible in a demo, and the rest of this guide is a procedure for testing them one at a time.

Use the following packet to make a pilot reviewable. These are proposed evidence requirements; they are not universal purchasing standards or measured predictors of commercial success. Each row is a thing the supplier hands over, the check the buyer can perform on it without specialist help, and the question that check settles.

The order matters. The task brief comes first because every later item is judged against it. The trial record comes before the final artifact because a perfect artifact from the third attempt means something different from a perfect artifact from the first. And operating assumptions come last because they are the part the buyer will have to reproduce alone.

Anthropic’s evaluation guide distinguishes a task from a trial and recommends multiple trials because outputs vary. It also separates a narrated success from the final state of the environment. Those distinctions make a useful foundation for a buyer-controlled pilot: one task run once tells you what happened once, and an agent that says it finished has told you what it believes, not what the system now contains.

Choose cases that represent the work you expect to assign. Keep a record of how they were selected. A pilot made entirely of easy, clean inputs cannot answer how the system handles incomplete attachments or conflicting instructions, and those are the inputs production will eventually send.

Decide which failures block adoption and which are acceptable with review. A wrong recipient may be disqualifying even when most routine cases pass. Report the counts and conditions rather than turning a small, selected sample into a general reliability percentage. For example, “eleven of twelve passed” is a fact about twelve cases you chose; “92% reliable” is a forecast about cases you have not seen, and the pilot did not produce it.

Record setup, retries, manual corrections and final review separately. If an engineer quietly repairs an export, the resulting file may be excellent, but the operating model includes that engineer’s work. The buyer should know whether the proposed service includes such support, because after purchase the engineer is either on the invoice or absent.

Ask what happens when the agent encounters a new exception. Is there a named escalation owner? Is the partial result preserved? Can the buyer understand the state without replaying a long chat? The tool-error decision table can help define these cases, and the handoff of a half-finished task to a person is covered in our guide to who owns unfinished work.

For a cost comparison, collect a comparable baseline and count the setup, intervention and review work observed during the pilot. The comparison is between the whole operating model and the current one, not between the model’s per-task price and a person’s hourly rate.

Anthropic’s harness report describes cases where code changes and limited checks did not establish working end-to-end behavior. It is a vendor engineering report, not evidence about the supplier you are evaluating, but the lesson transfers: the agent’s own checks can all pass while the thing the user needed does not work.

For a content agent, open the exported artifact. For a coding agent, run the agreed user action. For a research agent, inspect consequential citations. Choose the acceptance check for the deliverable rather than asking for the same dashboard screenshot in every pilot. The screenshot is the supplier’s view of success; the acceptance check is yours.

The file acceptance reference makes this concrete for documents. A downloadable deck that cannot be edited may be a successful export and an unsuccessful delivery, and only the person who has to edit it next week can tell the difference.

A pilot run by the book can still produce the wrong decision, and the failures are worth naming because each one feels like rigor at the time.

The four patterns below are illustrative. The curated sample: the buyer chose the cases, but chose them from the files that were easy to find, which are the clean ones. The moving rule: acceptance was agreed up front, then quietly relaxed when the first case failed on a technicality everyone agreed was minor; by the sixth case the rule has moved three times. The supplier’s environment: the pilot ran on the supplier’s infrastructure with the supplier’s access, and nothing about the buyer’s permissions, network or data residency was exercised. The absent exception: no case in the sample had a missing input or an ambiguous target, so the exception row of the packet was filled in from the supplier’s description rather than from observation.

Each is detectable from the packet. The task brief shows how cases were selected. The trial record shows whether the rule changed. The operating assumptions show whose environment ran the work. And an exception row that cites no case is an empty row.

Write the decision in one of three forms: ready for the agreed scope, continue a limited pilot to resolve named gaps, or not suitable for this task. List the evidence supporting that decision and the conditions that would invalidate it, such as removing required human review. A decision with its conditions attached can be revisited; a decision without them can only be defended.

For selecting trial cases, the guide to testing on your own work provides a related framework. Keep contractual and procurement review separate from the empirical claim that the agent can complete the job; a good contract does not make a weak pilot stronger, and a strong pilot does not need the contract to say the agent works. Buyers who want the pilot designed, run and written up as a decision record can do that with our AI transformation practice.