A coding agent edits a second agent's harness one patch at a time, and a hidden task set decides which patches stay. With the model weights fixed, three enterprise benchmarks gain 20–35 points, cost per task falls, and the evolved harness transfers unchanged to other models.
An agent is a model plus a harness. The harness is the code around the model: the system prompt, the tool definitions, the argument cleaning, the skills, the subagents, and the loop that decides when the agent stops. In an enterprise environment the harness often does not match the environment. Tool schemas omit the rules of the system. A ticket update must change the priority field and the impact field together. A strict MCP server (Model Context Protocol, the standard interface that exposes tools to an agent) rejects a call that contains an explicit null. A finance workflow must resolve "last month" against the sandbox clock, not against the model's idea of today. The model cannot learn these rules, because its weights are fixed. Engineers repair them by hand, one failed run at a time.
The authors call this model–environment mismatch. They ask two questions. Can a search procedure repair the mismatch automatically? Do the repairs hold on tasks the search never saw?
StarHarness treats the harness as a code base and runs an optimizer over it. The optimizer is a second coding agent, the proposer. It reads the current harness and the traces of failed runs, and it proposes one patch at a time. A loop tests each patch and keeps only the patches that improve a hidden score. The proposer runs inside Oh My Pi, a variant of the Pi coding harness. The harness under evolution is Stirrup, the open agent harness that Artificial Analysis uses for its evaluations. GPT-5.4 is both the model inside Stirrup and the proposer that edits Stirrup.
The paper calls the current best harness the frontier. Each iteration asks the proposer for one bounded patch, validates it, scores it on a hidden task set, and keeps it only if the hidden score improves. A persistent ledger carries scores, per-task outcomes, accepted hypotheses, and discarded attempts between iterations.
Two cheap gates run before the hidden evaluation. A validator checks that the patch stays inside the editable directories, that imports resolve, and that one task still runs. The validator also rejects a patch that branches on task identifiers, hard-codes answers, copies verifier text into prompts, or reads hidden state. Every accepted patch must describe reusable environment behavior, not one answer. Then the proposer names one search task that the patch should fix, and the loop runs only that task. If the task does not change from fail to pass, the loop rejects the patch without the full evaluation.
The key design is the task split. First the loop runs the baseline harness on every task and labels each with its failure mode (wrong tool, lost context, missing evidence, or premature conclusion) and its score. It samples a small evolution pool so that every failure mode is present. The pool is split in two. The search set is visible to the proposer, which reads its traces and outcomes. The selection set is hidden: the proposer learns only whether its mean score went up or down. The remaining tasks form the holdout set, which never affects any decision.
The authors run two search modes. Hill climbing keeps one frontier and accepts only strict improvements. Tree search keeps several candidate harnesses with parent links, so the proposer can debug a failed candidate or merge two compatible ones. On one benchmark they ran tree search first and hill climbing second, so the paper gives no fair comparison between the two modes.
The three benchmarks are ITBench SRE (40 Kubernetes root-cause scenarios), EnterpriseOps-Gym ITSM (103 ServiceNow workflows, graded by SQL checks on the final database state), and AutomationBench Finance (100 finance workflows across 47 simulated SaaS applications, graded by assertions on the environment state).
With GPT-5.4 fixed, the evolved harness raised the full-benchmark score from 40.0% to 75.0% on ITBench, from 23.3% to 43.7% on EnterpriseOps-Gym, and from 57.1% to 83.2% on AutomationBench. Cost per task fell by 17%, 53%, and 29%. On EnterpriseOps-Gym the pass rate on the benchmark's verifier checks rose from 34.5% to 72.8%. On AutomationBench, guardrail violations fell from 33 to 4. The search accepted only 21 patches in total: 4, 12, and 5.
Read the held-out numbers first. On the tasks used for evolution, GPT-5.4 gained 45.0, 22.0, and 23.0 points. On held-out tasks it gained 31.7, 15.1, and 29.3 points. Two of three benchmarks show a smaller gain on unseen tasks, so part of the improvement is fit to the evolution pool. The gain is still large.
The frozen harness, evolved with GPT-5.4 and never re-evolved, improved every other model the authors tried. Qwen3.5-27B went from 25.6% to 70.0% on ITBench. GPT-5.4-mini went from 29.6% to 70.0% on AutomationBench. The strongest model, GPT-5.5, gained the least on EnterpriseOps-Gym, at 10.7 points. Weaker models gain more, which fits the mechanism: the harness supplies knowledge the model does not have.
Against GEPA, a prompt optimizer applied to the Pi harness, StarHarness leads by 13.8, 22.3, and 17.6 points. The authors state that this comparison is descriptive only, because the two systems differ in prompts, tools, and architecture. The Codex coding harness is a closer reference. On EnterpriseOps-Gym it scores 41.7% against 43.7% for StarHarness, because Codex ships with built-in MCP schema normalization that already covers part of what StarHarness learned by search.
More caveats apply. The authors cannot isolate the contribution of any single patch. The paper does not report the cost of the search itself, only the per-task cost of the finished harness. The authors work at ServiceNow, and EnterpriseOps-Gym is a ServiceNow benchmark. The AutomationBench subset and harness differ from that benchmark's own paper, so those scores are not comparable to published ones.
The accepted patches fall into three groups. Interface repair: strip null, empty, and placeholder arguments before a tool call reaches the server; narrow and enrich tool schemas; replace raw spreadsheet edits with structured row operations. Environment conventions: write the implicit rules into the prompt, such as "update priority and impact together" and "anchor relative dates to the sandbox clock". Operational knowledge: move repeatable work out of the model's reasoning into deterministic tools, such as a forensics overview that ranks candidate root causes, and date and finance calculators. The trace analysis links these patches to fewer false-positive diagnoses, shorter workflows, and fewer unsafe executions.
You can run the protocol by hand. Collect the traces of your agent's failed runs and label each with a failure mode. Split the failing tasks into a set you inspect and a set you never inspect. Propose one change to the harness, and accept it only if the hidden set improves. This is a regression suite for a harness, with a hidden partition so that you do not tune to your own test cases.
The patches are also a checklist. Add an argument sanitizer in front of strict MCP servers. Write the conventions of your system into the system prompt as explicit rules. Replace arithmetic and date logic with tools. The automated loop is on GitHub at ServiceNow/StarHarness. Keep the selection set hidden from the proposer. That one separation is what makes the held-out numbers meaningful.