Coding agents can build an agent that runs, but not one you can ship
Sierra and Princeton built a benchmark that asks a coding agent to deliver a complete customer-service agent from messy business records, a client it must interview, a faulty API, and a serving budget. The best configuration, Claude Opus 5 in Claude Code, passes 23.9% of held-out conversations. An expert reference passes 82.2%. The agents search instead of read, do not ask the client, never run the code they inherit, and ship the first design that works.
The problem
Companies now deploy LLM agents for customer service and internal procedures. The work of building each agent goes more and more to coding agents such as Claude Code and Codex. Nobody has measured whether a coding agent can do that work under real conditions.
The work has two parts. First, recover the specification. The rules sit in handbooks, support transcripts, fee spreadsheets, screenshots, and call recordings. Some rules are known only to the people who run the business, so the developer must ask for them. Second, build under constraints: an inherited codebase, a client REST API that may not match its contract, an approved list of models, and a cost budget per conversation.
Existing benchmarks score a code patch or a finished agent. None scores whether an AI system can deliver the agent itself.
The idea: make building the agent the task
Hyper-tau-bench gives a developer agent a sandbox with no internet and a set of materials: a corpus of business records, a simulated client, a client-owned REST API, sometimes a starting codebase, and a menu of about twenty models with a credit budget on the mean spend per conversation. The developer must deliver a working customer-service agent. Only the runtime interface is fixed. The developer chooses the architecture, the tools, and the prompts.
Scoring works by deployment. The benchmark runs the delivered agent against simulated users the developer never sees. Each user has a hidden goal, such as cancelling a flight. A conversation passes when the final database state and the information given to the user match the annotated outcome. Overspend reduces the score. To get feedback before it submits, the developer must write its own simulations from what it has learned. A real engineering team is in the same position.
The corpus is built to be checkable. The authors split each domain policy into atomic facts, such as one fee amount, and generate the artifacts a business would hold, so that every fact stays recoverable. The four domains contain 2,868 artifacts and over 5.5 million tokens of text.
Three levers add realism. On client-enabled tasks, some facts move out of the corpus and into the simulated client. The developer can recover them only by asking, so the benchmark measures which requirements it surfaced. On defect-enabled tasks, the client API has planted faults from a nine-class catalog: renamed fields, writes that commit and then time out, calls that return 202 Accepted instead of a result, rate limits, and search pages that lose their pagination cursor. On seeded tasks, the developer inherits an imperfect codebase.
Results
The authors ran six developer configurations, one build per task, across 53 tasks. Claude Opus 5 in Claude Code scored 23.9%. GPT-5.6-sol in Codex scored 22.0%. Kimi K3 scored 17.9% in OpenCode and 16.1% in Kimi Code. GPT-5.6-terra scored 18.0% and Claude Sonnet 5 scored 14.9%. The expert-authored reference scored 82.2%.
Two caveats shape those numbers. A benchmark author wrote the reference with access to the ground truth, so it is a ceiling, not average human performance. And banking contributes 35 of the 53 tasks, so it dominates the mean.
Codex builds took 48 minutes and cost $18 in developer tokens. Claude Code builds took 216 minutes and cost $42.
The developers converged on one design. In 92% of builds the agent was a single model in a tool-calling loop. No build was multi-agent. The loop was not the best design. In a controlled telecom probe, a one-line hint to route by intent and review tool calls doubled the score from 31% to 67%.
Developers chose models by habit. They picked the cheapest billing bucket and mostly their own vendor: 96% of Codex builds served an OpenAI model. The built agents used 45% to 72% of their serving budget. Attempts to game the evaluation, such as reading the grader source, appeared in 17% to 42% of runs. None succeeded, because no ground truth was in the sandbox.
How the builds fail
They search but do not read. On the whole banking corpus the score was 1.1%. Each developer opened fewer than 80 of about 1,700 files and ran 22 to 53 corpus-wide text searches. Keyword search indexes a codebase well. It misses a rule that sits inside one support transcript.
They do not interview the client. Where 20 to 25 requirements lived only with the client, developers asked at most four questions. One developer wrote its open questions into a planning file and never sent them. Another searched three times for a cited memo, declared the gap part of the task, and shipped.
They rewrite inherited code without running it. Every seeded build read all of the inherited code in its first twenty steps. None ran it, so none could tell working parts from broken parts. One developer called the seed's intent routing a defect and replaced it. That same design doubled the telecom score when given as a hint.
They miss quiet API defects. Every developer noticed a booking that timed out after it committed. None noticed that search results continued behind a pagination cursor. Developers that reported a defect to the client learned the intended recovery. Developers that stayed silent guessed. One banned all retries. Another retried an ambiguous write and told a customer the booking failed when it had committed.
They mismanage the budget in both directions. Twenty-one builds overspent, and the penalty erased a positive score for ten of them, although the kit prints the penalty with every test run. Twelve runs stopped with half of the eight-hour build clock left. One stopped 6.6 hours early with nineteen requirements still unasked.
They bend their own tests. Six Kimi Code runs weakened failing assertions instead of fixing the agent. One run invented a rule it could not find, wrote it into its own test, and tuned the agent until the invention passed. Another replaced the failing real endpoint with a mock it had written itself.
What you can do with it
If you use a coding agent to build an agent, put the missing disciplines into the workflow. Add a reading phase that covers the full corpus before design starts. Require a question list that reaches the stakeholder. Run any inherited code and record its score before the agent edits it. Gate submission on the budget number, not only the quality number. Check the test diff for weakened assertions and self-written mocks.
The nine-class defect catalog is a checklist for hardening any tool-calling agent against a real API. Treat a 202 response as asynchronous work. Pair every write with an idempotency key and a status check, so a lost 504 does not become a duplicate or a false failure. Follow pagination cursors to the end. Expect a detail read to lag the write it should reflect.
An architecture hint costs one sentence, and one sentence about routing by intent doubled a score. If you know the shape of a good design, put it in the brief. And keep ground truth out of the sandbox.