Where Agents Can Check Their Own Work, and Where Not

Agent self-verification has a ceiling, and it is set by the output modality — not by how capable the model is. An agent iterating on a web page can read its own markup, its console errors and a screenshot of the result, so the fix-check-fix loop closes without you. The same agent producing a 3D scene, a rendered video or a physical behaviour is working partially blind, because the artifact is no longer something it can read.

That distinction matters more every month, because agents are moving up exactly this ladder. Browser and computer-use tooling made web output self-checkable; video-as-code frameworks put film in the repo; robotics research is wiring vision-language agents into simulation pipelines. Teams that treat “the agent verifies its own work” as a global property — rather than a per-modality one — end up trusting checks that never actually ran.

This guide builds the ladder rung by rung: what the agent can inspect at each level, what instrument to hand it so it can look, and when the honest answer is that you are the checker. Every capability claim below traces to vendor documentation or published research, and we flag which numbers are self-reported.

An agent can only self-correct what it is able to inspect. That sounds obvious written down, but it cuts against how most teams reason about agents in practice. We benchmark models on capability — can it write the code, generate the scene, produce the film — and then quietly assume the verification step inherits the same score. It does not. Verification depends on whether the artifact the agent just produced is something it can read, and readability falls off a cliff as output moves from text toward the physical world.

Our working model is a four-rung ladder. Each rung up, the artifact loses a form of readability: first the direct text view, then a spatial axis, then the time dimension, then appearance itself as the criterion. The table below is the spine of this piece — our own synthesis, built from the sourced tooling and research facts in the sections that follow.

Web output is the easy case for one structural reason: the artifact is text all the way down. Markup, computed styles, console output, network responses — an agent reads them the same way it reads source code, with no perceptual step in between. Add a screenshot on top and the loop closes: the agent sees both what it changed and what the change looks like, and can compare either against a mock. We covered the iteration technique for this rung in screenshot-driven UI development with vision models; this section is about what makes the rung special, because nothing above it works this well.

The tooling is production-grade today. Two instruments matter most for teams on the Claude stack, and equivalents exist across vendors — see our computer-use agent comparison matrix for the wider field.

Two operational details from the vendor docs are worth knowing. First, verification-shaped calls are privileged: in Claude Code's plan mode, browser actions that only read state — reading the page, extracting text, checking console and network messages, taking a screenshot — run without a permission prompt, while anything that changes state (clicks, typing, navigation) asks first. The architecture itself treats looking as safe and acting as consequential, which is exactly the right mental model for this rung. Second, reading is not infallible even here: Anthropic's own limitations page says plainly that Claude “might make mistakes or hallucinate when outputting specific coordinates while generating actions,” and notes that scrolling can be unreliable in some applications. Anthropic also characterizes Sonnet 4.6 as more mechanically precise at clicking than Opus 4.6, with Opus 4.7 narrowing that gap — a vendor-stated comparison, not an independent benchmark.

The docs carry one more practical recommendation: for computer use on Sonnet 4.6 and Opus 4.6, Anthropic advises medium thinking effort as the best accuracy-to-cost ratio and advises against maximum effort, which in its testing adds token cost without improving accuracy on UI tasks. Verification loops run many iterations, so the per-iteration setting compounds — a point we return to when this rung shows up inside production web development workflows.

One rung up, the inspection primitive breaks. A screenshot of a 3D scene is a single projection of a structure with depth, occlusion and unboundedly many valid viewpoints — the pixels the agent can “read” have collapsed the very dimension it needs to judge. Is the mesh actually the right shape from the angles it is not currently looking at? Is that geometry intersecting behind the visible surface? No amount of squinting at one frame answers those questions, for a model or for you.

The research record backs the intuition. ViewSpatial-Bench, a 2025 benchmark from Li et al. covering five multi-perspective spatial-localization task types, reports that current vision-language models “excel primarily at egocentric spatial reasoning” — from the camera's own perspective — “but fail to generalize to allocentric viewpoints,” meaning reasoning from another entity's or another camera's point of view. The authors also report that fine-tuning on their multi-perspective spatial dataset produced an overall performance improvement of 46.24% across tasks — a relative improvement on the paper's own five-task suite, self-reported by the authors rather than independently audited, but a strong signal that the gap is trainable rather than fundamental.

Why does the gap exist at all? The framing shared across several independent research groups' 2025-2026 work — 3DSRBench, All-Angles Bench, OmniView-Space, and an ICLR 2026 paper on cross-view reasoning — is that vision-language models are trained predominantly on 2D image-text pairs scraped from the web, which carry no explicit 3D spatial annotation. Models lean on single-image shortcuts because their training data never made them pay for it. Whatever the eventual fix, the operational conclusion for today is the same: one screenshot of a 3D scene is not a check.

What to hand the agent, then, is more views and more structure. Render multiple viewpoints — not one hero angle — so consistency across projections becomes checkable. Better still, expose the scene graph, transforms or mesh statistics as structured data. That move is worth stating as a principle: the fix for rung 2 is converting as much of it as possible back into rung 1, where the artifact is text and the agent can read it. What cannot be converted — the aesthetic judgement of a form from an unrendered angle — stays with you.

Video stacks a new dimension on top of the 3D problem: time. Checking a video is not checking an image — it is reasoning over a sequence of frames, where the properties that matter (motion pacing, continuity, whether an entrance lands) exist only across frames, never inside one. An agent can sample stills and diff consecutive frames, but “the animation feels right” is a judgement about the deltas, and today that reasoning is the weakest link in the loop.

The video-as-code movement makes this rung unusually concrete, because the source is fully readable even though the output is not. Remotion — the framework we examined in depth in Video as Code: Remotion and the Agent Feedback Gap — treats each frame as a React component: the current frame number arrives via a hook, the renderer drives a Puppeteer-controlled browser to produce the image sequence, and a bundled FFmpeg encode (no separate install since v4.0) stitches the frames into the final file. The entire film lives in git and typechecks. And none of that tells the agent whether the motion matches the brief — the render exiting cleanly is a statement about the pipeline, not the picture.

Remotion's own definition doubles as the verification problem statement: if a video is a function of images over time, then checking it means evaluating the function, not one of its values. When we audited the available comparison tooling for the Remotion piece, the tools we checked fell into two families, neither aimed at intent: static-frame pixel comparators (twenty-twenty, jest-image-snapshot) judge one frame at a time against a known-good reference, and encoding-fidelity metrics (SSIM, PSNR, VMAF) score a processed video against its pre-encoding source. Neither answers “does this animation do what the brief asked” — and a newly generated animation has no known-good reference to diff against in the first place. That finding is scoped to the tools and metrics we examined, not a claim that no such tool exists anywhere.

So the practical instrument for this rung is a harness you assemble: sample frames at meaningful intervals, diff consecutive frames so motion becomes a readable delta, pipe in the render logs, and have the agent narrate what it believes each transition is doing before you watch the cut. The narration step matters — it converts the agent's implicit claim (“this works”) into an explicit one you can falsify in a single viewing. Prior research supports the general pattern: iterative visual self-refinement, where a model critiques its own generated output and revises, predates agentic video entirely — Idea2Img (ECCV 2024) demonstrated the generate-critique-revise cycle for image generation, validated by a user preference study rather than an automated benchmark. The cycle extends to frames; what has not yet caught up is the temporal reasoning between them.

At the top of the ladder the criterion itself changes. For every rung below, “correct” ultimately cashes out in how the artifact looks — even video is appearance over time. Physics is different: correctness means the behaviour holds under forces, constraints and time. Does the linkage bind? Does the gripper's grasp survive the object's actual mass distribution? A rendering of the outcome — however photorealistic — cannot settle those questions, because appearance no longer carries the information the check needs.

This is why the robotics and simulation communities do not try to look harder; they build worlds to run. A representative example from July 2026: Agentic Real2Sim (arXiv 2607.19190, Chen et al., 24 authors) uses vision-language agents to orchestrate a pipeline of deterministic specialist tools that convert a real-world recording of an object-robot interaction into what the authors call a “simulatable episodic twin” — preserving observations, geometries, robot interactions and object states — automating mesh cleanup, coordinate alignment and cross-tool integration steps that previously took manual effort. The authors report testing across rigid objects, deformable materials and humanoid motion, and that open-weight VLMs achieved comparable conversion success rates to frontier models at lower cost. As with the 3D rung, these are the researchers' own reported results, not yet independently reproduced at the time of writing.

Note the architecture, because it is the ladder's endgame in miniature: the agent's role is not to perceive correctness but to build and operate the instrument that measures it. The same direction shows up across the field — NVIDIA's 4D digital-twins workshop at CVPR 2026, and a digital-twin framework validated on a standard robotics manipulation task — all premised on the point that a photo or video of a physical outcome cannot certify that a policy or design behaves correctly under different conditions. The check requires running the simulation forward, or a physical trial. Hand the agent a simulator with scored outcomes, or a test rig that emits numbers, and you have moved the problem back down to rung 1 — the score is text. Without one, no amount of model capability substitutes for the missing instrument.

The ladder converts into a per-rung operating decision. Before delegating any output modality to an agent, answer one question: what will it inspect, concretely, when it claims the work is done? If the answer is “nothing beyond its own generation,” you have not delegated the work — you have delegated the first draft and kept the review, whether you planned to or not.

Two companion practices multiply the value of whatever instrument you hand over. First, pair the instrument with an explicit definition of done — an agent with a browser but no stated bar will verify against its own guess of the requirement. We break that discipline down in Define Done, Not Effort, a companion piece in this series. Second, know which axis of verification you are buying. The ladder is about perceptual self-inspection — an agent examining its own artifact. Consensus methods, where several models review the same reasoning, address a different failure class entirely, and agreement is not proof — we covered why in our cross-model review method. Neither substitutes for the other: three models agreeing about a video none of them can watch is still a guess.

It is worth being precise about how much measurement stands behind this ladder, because the honest answer is: strong on the bottom rung, directional above it. The clearest quantified result comes from web agents — the rung where verification is easiest to build and easiest to score.

Read the pattern in where the numbers live, because it is the trend that matters: the quantified wins concentrate on the rung where output is readable and scoring is cheap, and thin out as the artifact gets less inspectable. That is not a coincidence of research fashion — it is the ladder measuring itself. Benchmarks need ground truth, and ground truth is exactly what gets expensive as you climb. The 3D and physics results that do exist are self-reported by their authors on their own benchmarks, which is normal for young research areas but means the error bars are wider than the abstracts read.

Equally important is what we did not find. Among the sources we reviewed for this piece, no independent benchmark quantifies relative verification difficulty across the 3D and physics rungs — nothing that would let you say “rung 3 is N times harder than rung 2” with a straight face. That is why this piece argues the ladder qualitatively and attaches numbers only to the narrow claims they actually support. Anyone selling you a cross-modality verification score today is quoting a number that, as far as we could establish, does not exist.

The projection we would stake: over the next several quarters, the gap between rungs narrows from the bottom up — not because models suddenly perceive depth or time, but because toolmakers keep converting unreadable artifacts into readable ones. That is already the shape of every win on the board: browser tools turned pixels into DOM-plus-screenshot pairs; scene graphs turn geometry into text; simulators turn behaviour into scores. Expect frame-harness and multi-view tooling to become standard agent equipment the way browser tools did, and expect the marketing to outrun the instrumentation the whole way — “the agent verifies its own work” will keep being said globally while remaining true per-rung.

For teams, the budget conclusion follows directly. When you scope agentic work, price the instrument alongside the agent: a dev server and browser tooling for web deliverables, a multi-view render step for 3D, a frame-sampling harness for motion work, a simulator or test rig for anything physical. Where the instrument does not exist or is not worth building, put the human review on the schedule explicitly — unplanned review is still review, just late and resentful. This is how we structure delivery in our own AI transformation engagements: the verification loop is designed per output modality before any agent writes a line, because that decision — not model choice — determines how much a person has to stay in the loop.