AI Agent Test Data: What to Fake and What to Keep Real

Fake the inputs that make a test safe and repeatable, while keeping the behavior under examination real. If an agent is supposed to save a report, a test that replaces the save operation with a success message can verify the agent’s response to that message. It cannot establish that a report was stored.

The reference below covers sixteen selected test cases. Use it to ask what each passing test actually demonstrates. “Real” here means the relevant implementation or boundary is exercised in a controlled environment. It does not mean using customer records, sending live messages or spending money to make a test look authentic.

A mock is a controlled replacement for part of the system. A fixture is a prepared input or starting state. Both are useful. The table identifies what must remain exercised for the named test to mean anything, then names the claim that still needs separate evidence. Rows are choices for test design, not a ranking from bad to good.

Playwright’s API-mocking guide shows an intercepted request fulfilled with a custom response without calling the API. It also distinguishes modifying a real response from replacing the request altogether. That difference determines whether a passing browser test observed the remote system.

Playwright’s testing guidance recommends user-visible checks, isolated tests and controlled data. It also recommends mocking third-party dependencies. These are useful ways to test the application you own. They do not turn a substituted dependency into evidence of a live integration.

The table extends that boundary into agent tests as an editorial design reference. It does not claim Playwright implements every listed substitute or that all agent tests belong in a browser. A file-processing agent and a scheduling agent need different controls, even if both produce a final status message.

Imagine a hypothetical agent that turns a short input file into a report and delivers it to a review folder. Synthetic names and a tiny example file can be appropriate. Keep the actual transformation and file-writing code in the test, then reopen the output from the controlled folder. That checks more than asking the agent whether saving succeeded.

If the remote upload is mocked, label the result “local report generation checked; remote storage simulated.” A separate authorized check can use a disposable destination and confirm the uploaded copy. Inspect content as well as existence: an empty file at the expected path does not establish report generation.

The delivery-state reference helps describe the resulting boundary. The file-output acceptance reference supplies the checks for the actual copy. A passing local test should not silently acquire a claim about recipient access.

Synthetic data becomes misleading when it removes the difficulty the tool must handle. A parser expected to accept names with accents should not be tested only with short plain-English names. A calculation that handles missing values needs deliberate missing cases, not a spreadsheet where every cell is populated.

Preserve relationships as well as individual values. If two records are meant to refer to the same entity, give them a consistent synthetic identifier. If a duplicate is intentional, label why it is present. Otherwise a test can pass because the fixture accidentally avoids the business rule under examination.

Our own-traffic evaluation guide addresses whether tasks represent intended use. This reference addresses what gets replaced within those tasks. Representative examples still need clear substitution boundaries; realistic-looking records alone do not make an integration real.

For each test, write: claim, fixture, substituted boundaries, implementation exercised, observed result and remaining check. Include the test version so someone can tell whether a later run used the same setup. This note is more useful than a single “all tests passed” message when the reader must decide whether work is ready.

Do not invent a universal minimum number of examples. Choose cases from the behavior you promise and the failures you need to distinguish. A controlled expiry response can examine recovery logic; an authorized expired credential in a test environment examines whether the real boundary produces the expected denial. Neither alone estimates how often expiry happens.

Keep expected outcomes separate from the agent’s own assertions. For a transformation, use an independently specified expected value or a property the output must satisfy. For a write, inspect the destination through a separate read path where practical. The test should have a way to reject a plausible explanation of a result that never occurred.