A controlled study of 1,116 agent-built web apps shows that a verification tool improves the shipped software only where the tool can observe the failure. A one-bit boot probe removes almost all launch failures at the lowest cost in the study. A linter changes nothing. Screenshots help only on visible defects, and the evidence for even that is weak.
A coding agent ships files, not conversation. Harness builders decide which tools the agent gets to check its own work: a linter, a shell, a screenshot tool. The author calls this set the agent's verification surface. Prior studies compare whole harnesses, so they cannot say which tool contributes which part of the quality. This study varies only the tool list and holds everything else fixed.
The author built a minimal coding agent. The system prompt, the task text, and the loop are identical across all runs; a hash of every prompt component proves it. Only the tool list changes. Six models (three Claude models, gpt-5.5, gemini-3.1-pro, grok-4.3) each implemented seven web applications — a message board, a kanban board, a seat-booking app with race conditions, a calendar view, a metrics dashboard, a modification task on unfamiliar code, and a log viewer that must scroll smoothly over 100,000 rows. The result is 1,116 applications.
The study grants five core tool sets, from nothing to everything. No tools: the agent writes everything in one pass. Linter: static checks only, no execution. Boot probe: one tool that installs dependencies, starts the server, and reports one bit — does it listen. Shell: unrestricted command execution. Shell plus screenshots: the agent can also see a rendered image of its page. Three more tool sets isolate causes: screenshots without a shell, a shell that unlocks only after the agent declares it is done, and a tool that only runs tests the agent writes itself.
A human graded every application against a rubric frozen before any run existed; the scores below are points on it. The grader saw the runs shuffled, with model and tool set hidden; a blind regrade of a 10 percent sample four weeks later reproduced 98.8 percent of verdicts. Automatic probes stress-tested the API behavior: eight simultaneous requests for one theater seat must produce exactly one winner; a process kill and restart must not lose data. An app that never starts scores zero on everything — dropping such runs would reward the tool sets that produce the most of them.
The boot probe gives the most for the least. With no tools, about one build in seven never launches (86.5 percent start and serve requests). The single boot bit lifts that to 99.5 percent. It comes within 2.5 points of the full shell on the API-tested tasks: ten of the twelve points that separate no-tools builds from shell builds arrive with the boot probe alone. And it is the only tool set that costs less than no tools — 214 thousand median tokens against 262 thousand. The agent finishes in fewer steps, and because the whole history is re-sent every step, fewer steps means fewer input tokens. It also has the smallest run-to-run cost spread in the study.
The linter adds nothing. The linter tool set started slightly fewer builds and scored slightly lower than no tools at all; both differences are too small to mean anything. The failure is concrete: ten builds in the study shipped a frontend file that could not parse, and five of them came from the linter tool set — the one tool set equipped to catch exactly that. In one logged run the model called the linter once, received "Missing script: lint" because it had never set the tool up, and finished without checking again. The agent skipped the linter entirely in 23 percent of the runs that had one. Granting a tool is not using it.
The shell adds behavioral depth at 2.35 times the cost. The shell's remaining points over the boot probe appear only on the hardest probes: simultaneous-request races, kill-and-restart persistence, ordering under load. Code volume stays flat across tool sets (716 to 800 mean lines), so the extra tokens go into checking and re-running, not into more code.
Screenshots give the least clear result. On the two visually hard tasks the screenshot tool sets score highest, and the pooled gain over the shell is +6.9 points. But this is the one main effect that fails the correction for testing many effects at once, and the per-model split weakens it further: the largest gain (42.9 points) belongs to grok-4.3, which took one screenshot, while heavy screenshot users mostly gained nothing. On the log viewer, where the failure is stutter that no still image can show, screenshots add nothing — the study built that task as a control, and it behaved as designed.
Self-written tests are not a substitute. The tool set whose only check was running its own test suite scored 16 points below the shell. The agent wrote tests, watched them pass, and finished early — after 22 steps against the shell's 49. The author's diagnosis: a model's own tests encode its own reading of the task, so a misunderstanding passes its own tests and ships.
The defect catalog shows failures that sit between the tools. One calendar rendered pixel-accurately, passed its own positioning tests, and ignored every click — an invisible overlay sat on top of the page and captured every mouse event. Only clicking finds this, and the passive screenshot tool cannot click. One dashboard's stylesheet overrode the mechanism its own JavaScript used to hide the "backend unavailable" banner, so a healthy app permanently reported itself down. One server stored dates as text and crashed on the first request that did arithmetic on them — the boot probe passed it, because the server started fine.
A further finding: which defect a build carries is predicted better by which model wrote it than by which tools it had. One data-directory bug appears in 41 percent of claude-4.6-sonnet's builds and near zero percent of grok-4.3's. Seven of ten structural choices (directory layout, module system, how the frontend calls the API) follow the model, not the tools. The strongest model gained nothing from any tool; the weakest gained 37 points from a shell.
The transferable rule is reach: a verification tool improves the artifact only where it can exercise the way the application actually fails. That gives a checklist for anyone configuring an agent harness. First, add a boot probe: it removes the largest failure class and lowers cost and variance at once. Second, do not assume a linter helps an agent the way it helps a human — measure it. Third, grant a shell when behavior under load matters, and budget 2.35 times the tokens for it. Fourth, add screenshots only when the work is visually hard, and do not mistake a passive screenshot for interaction testing. Fifth, do not rely on agent-authored tests as the only gate. Last, tune the verification budget per model: how much verification a model needs is a property of the model.
The limits are real: one author, one human grader (with a strong blind-regrade check), six models, seven small build-from-scratch tasks, and a screenshot tool that cannot click, so the study measures the low end of what sight can do. Both effects that involve human-judged quality are suggestive, not established. The paper is submitted to the IEEE and not yet peer reviewed. The full dataset, prompts, logs, and analysis code are public on Zenodo and GitHub.