Testing a Vision Model on Screenshots of Your Own App

Test a vision model on the questions your application actually needs it to answer. Can it read an error code, locate the correct Edit button, or recognize that a screenshot does not contain enough information? Score those abilities separately. A correctly located button is not evidence that the requested task was completed.

The protocol below is a proposed method for building a small, inspectable evaluation. It does not report model accuracy or recommend a winner. Begin with privacy-safe fixtures, label the expected answers before calling a model and preserve failures rather than selecting only convincing examples.

Grounding means locating a target from a screenshot and an instruction. SeeClick’s ScreenSpot evaluation uses annotated target regions for that kind of question. It does not make static target location interchangeable with a completed application workflow.

The OSWorld paper uses execution-based checks of resulting computer state for interactive tasks. That is a different measurement. Preserve the distinction even when the same model participates in both evaluations.

Use synthetic accounts and invented records in a controlled environment where possible. Include ordinary screens, dense interfaces, menus, error states and repeated labels. Select cases from the intended role of the model; do not assemble only attractive screenshots or only known failures.

The ScreenSpot-Pro paper examines dense, high-resolution professional interfaces and explicitly separates grounding from planning and execution. Its findings motivate difficult cases; they do not establish how frequently those errors occur in your product.

Give every case a screen-family identifier. Crops and near-identical variants of the same screen should not accidentally appear on both sides of a tuning/evaluation split. Keeping examples for improvement separate from held-out examples makes it harder to confuse memorized cases with broader capability.

Review captures for names, URLs, tab titles, overlays and private information. If masking removes the context needed to answer a question, retire or relabel the case. Do not treat it as the same evaluation input after a material edit. Our screenshot evidence reference explains what a visible state can establish.

Write the question, expected answer and acceptable alternatives before seeing model output. Have another reviewer examine the label where practical. Resolve disagreements with a recorded reason rather than changing the label to match a convincing answer.

For a target-location task, annotate the region a valid click may occupy. For visible text, preserve meaningful case, punctuation and digits. For an ambiguous question, record why it is ambiguous. A screen with two identical icons may need additional context rather than a forced guess.

Use hypothetical cases such as these to design your own fixture set. They are not screenshots or model responses collected for this article.

Keep the original capture dimensions, viewport, device pixel ratio, zoom and crop coordinates. Then record any provider preprocessing and the supplied image. A point in a resized crop must be mapped back according to the documented coordinate convention before it can be scored against the original target.

A screenshot-only response and a model-controlled crop/search loop are different systems. If you compare them, report their different calls, inputs and budgets. Do not quietly give one candidate more opportunities to inspect the target.

Pin the model identifier, prompt, retry policy and output format used in the evaluation. An image-input capability flag does not establish that the model can locate small controls reliably. Provider limits and coordinate conventions require an execution-time check; this method does not assert one universal convention.

For capture and iterative UI critique, see our screenshot-driven development guide . Here the additional requirement is a frozen question and label that allow a fair comparison.

Count exact reading matches over eligible reading questions after a declared normalization rule. Count valid target points over eligible grounding questions. Keep invalid responses in the relevant denominator and report request failures and timeouts separately. A failed parse should not vanish from the result.

Report correct abstentions on unanswerable cases and unnecessary abstentions on answerable cases separately. A model that declines everything can appear cautious while being useless for the intended role. Likewise, an answer that confidently invents a truncated label needs a distinct failure category.

Break results down by text versus icons, screen family, target size and state when the sample supports that breakdown. Publish numerators and denominators. A broad percentage can conceal the one repeated-control error that matters most to your workflow.

Do not silently mix first-attempt results with retry-assisted results. If retries are allowed, identify the policy before testing and show the additional attempts. For a model-switch decision on full workflows, our own-traffic evaluation guide covers a separate kind of replay.

Agree on critical failure types and acceptance thresholds before measuring them. There is no universal pass mark here. A model intended to summarize a screen may face a different requirement from an agent allowed to act on account settings. Keep the permitted role connected to the evidence.

Record case ID, image hash, capture and annotation versions, prompt, model, raw response, parsing outcome, normalized answer and reviewer comments. Preserve the original failure when an annotation defect is corrected. Distinguish model errors from capture defects and coordinate-transform bugs.

No fixtures or model runs were created for this guide. To make a quantitative comparison, build the dataset, review its labels and execute the frozen protocol. If you need help connecting that evaluation to the application’s behavior, our AI transformation service can help define the acceptance scope.