Daily paper · Tuesday, September 1

Same model, 35 harness releases: bug-fix rate flat, tokens up 70%

cs.SE · Ben Sghaier, Li & Adams et al. · arXiv:2607.03691

A study of 35 releases of one coding-agent CLI, with the model kept the same, found no improvement in bug-fix rate. Later releases used about 70% more tokens per task. Every regression passed the project's CI.

The problem

A coding agent is two things: a model and a harness. The harness is the program around the model. It writes the system prompt, defines the tools, runs the tools, decides what the model sees on each turn, and decides when the loop stops. Claude Code, Codex, Gemini CLI, and Qwen Code are all harnesses.

Harnesses change very fast, and most of them auto-update. When an agent gets worse after an update, users usually blame the model. Nobody had tested whether the harness alone can change the results. This paper does that test.

Loop: the harness sends a prompt to the LLM, the LLM answers with text or a tool call, the harness runs the tool in the environment, appends the result to the context, and sends the next prompt.
Figure 2 from the paper: the control loop inside a harness. The harness controls every step except the model's own reasoning.

How fast harnesses change

The authors first mined five open-source harnesses (Gemini CLI, Codex, OpenCode, OpenHands CLI, Qwen Code) from March 2025 to February 2026, and compared them with VS Code and the GitHub CLI.

The four mature harnesses shipped 10 to 18 releases per week; VS Code shipped 0.8 and the GitHub CLI 0.6. The harnesses merged 13 to 34 commits per day, and about 30% of commits were bug fixes. Issues arrived faster than maintainers closed them: OpenCode had a backlog near 4,000. The authors call this pattern hyper-churn.

The experiment

The authors kept one model fixed and changed only the harness. The harness was Qwen Code CLI, a fork of Gemini CLI. They installed 35 consecutive releases from npm, from v0.0.10 to v0.10.3. The model was Qwen3-Next-80B-A3B-Instruct, an open-weight model that they hosted on their own servers. Self-hosting removed two sources of noise: silent model updates behind an API, and API rate limits.

Each release ran 50 tasks from SWE-bench Verified, a public set of real GitHub bug-fix tasks, sampled to match the benchmark's difficulty mix (20 easy, 25 medium, 5 hard). Each task ran twice with a 600-second limit. That is 3,500 agent runs. A task counts as resolved only if the agent's patch passes the repository's full test suite.

They measured three things per release: resolve rate, total tokens per task, and tool calls per task. The two runs of a task agreed on pass or fail 87.7% of the time, so the differences between releases are not random model noise.

Results

Resolve rate did not improve. Across 35 releases it moved between 23% and 39%, with no statistically significant trend. The mean was 30.5%. The best release was v0.0.14, one of the earliest.

Cost went up. The first nine releases averaged about 391K tokens per task. The latest releases averaged about 668K. That is a 70% increase with no gain in correctness. Tool calls per task moved between 6.9 and 14.3, with the same upward trend. The trend remained when each task was compared with its own average across all releases, so a few hard tasks do not explain it. The growth comes from the harness.

Schematic of the main result: across 35 releases the resolve rate stays inside a flat band between 23 and 39 percent, while tokens per task rise from about 391 thousand to about 668 thousand, a 70 percent increase Resolve rate (share of 50 tasks fixed) 0% 39% 23% best release: v0.0.14, one of the earliest all 35 releases stay inside this band; mean 30.5%, no trend Tokens per task 0 first 9 releases: about 391K latest releases: about 668K +70% v0.0.10 (oldest) 35 releases, same model throughout v0.10.3 (newest)
Schematic, not the paper's data points: the resolve rate never leaves a flat band while tokens per task rise about 70%.

Two mechanisms explain the cost. First, the initial prompt (system prompt plus tool schemas) grew about 8% between the earliest and latest releases. Second, later releases needed about 18% more model turns per task. The harness resends the full conversation history on every turn, so the larger prompt is paid again on each extra turn. Failed tasks were the most expensive: 2.7 times more tokens and 1.8 times more tool calls than resolved ones. A failed run loops through edit, test, and read without progress.

Why tokens multiply: on every turn the harness resends the system prompt, the tool schemas and all earlier turns, so each row is longer than the one above it; a bigger prompt block and more turns both raise every row What the model receives on each turn of one task Turn 1 prompt + tools Turn 2 prompt + tools turn 1 Turn 3 prompt + tools turn 1 turn 2 Turn 4 prompt + tools turn 1 turn 2 turn 3 Turn 5 prompt + tools turn 1 turn 2 turn 3 turn 4 extra turn Every request resends all earlier turns. The cost of a task is the sum of all rows. Later releases: prompt block about 8% bigger, about 18% more turns. Both raise every row.
The harness resends the full history on every turn, so a bigger prompt and extra turns both act on every row.

The case studies show how one release changes cost. v0.1.4 was the cheapest release: 217K tokens per task at a 26% resolve rate. v0.1.5 shipped one day later with the same resolve rate and 330K tokens per task, a 52% increase. Two pull requests caused it: one rewrote the glob, grep, and ripgrep tools; one changed how tool output is formatted for the model. v0.3.0 more than doubled tokens over v0.2.3 (259K to 600K) after it added a 14,550-line streaming I/O layer and 4,740 lines of translation code, some of which reached the system prompt. The opposite also happened. A chat-compression fix in v0.1.2 cut tokens by 26%, because the old code let the context grow without limit, and removing five unused CLI modules in v0.5.0 cut tokens by 13%.

All of these changes passed CI. The project has more than 300 unit, integration, and end-to-end tests. None of them runs the agent on a coding task or checks a token budget.

Which changes are risky

The authors correlated 22 release-level factors with the three metrics, with a statistical correction for testing many factors at once. Feature-heavy releases had higher resolve rates and higher token use. Fix-heavy releases raised token use without raising resolve rate. Releases made of a few large pull requests used fewer tokens than releases made of many small ones. Deleting code lowered token use. Release cadence had no measurable effect.

They then mapped every changed file to one of ten harness components. Two components were high risk. Large changes to the LLM provider layer, the code that converts requests and responses between the harness and the model API, predicted immediate drops in resolve rate; in one release, a change to the OpenAI converter coincided with a drop from 39.4% to 32.5%. Growth in context management, the code that decides what history the model sees, predicted worse token efficiency. Two components were consistently safe: changes to extensibility (hooks and plugins) correlated with better token efficiency, and security fixes correlated with fewer tokens and fewer tool calls.

Risk map of the ten harness components: changes to the LLM provider layer and context management were high risk, changes to extensibility and security were consistently safe, the other six were not singled out High risk Consistently safe Not singled out LLM provider layerrequest / response conversionbig changes: resolve rate drops Context managementwhat history the model seesgrowth: worse token efficiency Extensibilityhooks and pluginschanges: better token efficiency Securitysecurity fixesfewer tokens, fewer tool calls Orchestrator Tool system Session manager Persistence Configuration UI
The paper's ten harness components with the study's verdicts. Where a change lands is a better warning sign than how often the project releases.

Caveats

The controlled study covers one harness and one model. The harness was pre-1.0 for the whole period, and the model is an 80-billion-parameter open-weight model, not a frontier model; a different model may react to prompt growth differently. With 50 tasks and 35 releases, the correlation analyses have limited statistical power, and correlation is not cause. A task is either resolved or not; a partial fix gets no credit. Claude Code was excluded because its source is closed. The authors say their result shows that the harness matters, not that the specific numbers transfer to other agents.

What you can do with it

Treat the model and the harness version as one unit. When an agent's behavior changes, check the harness changelog before you blame the model. Record the harness version in bug reports and in any evaluation you publish.

Pin the version of your agent CLI. Upgrade on purpose, not by auto-update. Before you accept an upgrade, run a small fixed set of your own tasks against the old and new versions, and compare pass rate, tokens per task, and tool calls per task. Ten tasks run twice is enough to catch a 50% token jump.

If you build a harness, add a cost regression test to CI. A nightly run of a fixed task set with a token budget and a tool-call budget would have caught every regression in this paper. Apply extra review to changes in the provider layer and the context manager. Track tokens per task as a primary metric, because it can double while every test still passes.

Read the paper on arXiv ↗