Index the harness by behavior, not by file
Every change to an agent harness starts with a search problem: the request names a behavior, but the behavior is implemented in many places, and the repository is organized by files. Harness Handbook builds a behavior-indexed map of the codebase from static analysis plus LLM structuring, revalidates the map after every diff, and gives it to the planning agent. With the map, a mid-tier planner chose edit sites almost as well as frontier models did, and it used fewer planner tokens.
The request names a behavior; the repository stores files
An agent harness is the software layer around a language model. It constructs prompts, manages state, invokes tools, and runs the control loop. Coding agents such as Claude Code and Codex are mostly harness code. The model changes rarely. The harness changes constantly, because models, APIs, tools, and requirements change.
Each change starts with the same question: where is this behavior implemented? A modification request describes what the system should do. The repository is organized by files and modules. One behavior often depends on several code sites that are far apart: a step in one execution stage, a shared state field in another module, a branch that rarely runs. The authors name the task of finding all of these sites behavior localization, and they argue it is the main bottleneck when harnesses evolve.
Code search, repository maps, and indexes make code easier to inspect, but they organize information around the implementation, not around behavior. A coding agent with a limited context window explores the repository step by step. It finds some relevant sites and can still miss the scattered ones. An edit plan built from an incomplete site list produces an incomplete patch.
A handbook that maps behavior to code
Harness Handbook is a second representation of the repository, organized by what the system does. It is a three-level document tree. L1 is a system overview: architecture, execution model, major stages, global data flow. L2 describes one execution stage: its responsibilities, inputs, outputs, dependencies, and local state. L3 entries are the leaves: each describes one implementation unit and carries a locator that points to the source. A separate state-register view records state that crosses stage boundaries, so stages that are coupled through shared state stay connected even when their code is far apart.
A pipeline builds the handbook from the repository automatically. Phase one is deterministic static analysis with no LLM calls: it parses the code and extracts functions, signatures, call edges, and named external boundaries. Phase two uses an LLM to assign source units to execution stages. It has two modes: function-as-leaf, used when a trusted outline of the harness's execution stages already exists, and file-as-leaf, which infers the stage structure when no skeleton exists or the repository is too large for function-level entries. Phase three synthesizes the L1–L3 documents and validates every locator against the current source.
Two rules keep the handbook trustworthy. First, progressive disclosure: a reader, human or agent, moves from L1 down to L3 only when the task needs more detail. Second, alignment: every L3 locator must resolve against the current repository. If a locator does not resolve, the pipeline freezes that entry and excludes it from localization until it is refreshed. The repository stays the authority; the handbook is a guide, never a substitute. After every non-empty diff, a resynchronization step reparses only the changed code, matches old and new units with fingerprints that do not depend on line numbers, and updates the affected entries.
At modification time, a workflow named Behavior-Guided Progressive Disclosure (BGPD) uses the handbook to localize the behavior. It selects the relevant stages from L1 and L2, follows the state registers to stages coupled through shared state, picks the matching L3 entries, and expands the candidate set along the call graph. Then it opens the real repository, resolves each locator, and keeps only the sites that are still relevant. Those verified sites become the evidence for the edit plan.
Results
The evaluation uses two open-source harnesses, Codex and Terminus-2, with 30 modification requests each. Each request has one of three types: query (change an existing behavior), cross-file (add a capability that spans modules), or search-hostile (the relevant code is hard to find with keyword search). A planner agent that reads the repository but does not edit it, powered by DeepSeek-V4-Pro, produces an edit plan twice: once with handbook access and once without. The requests, model, repository, tools, and settings are identical in both arms. Three LLM judges (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) compare the two plans for each request on localization, scope control, and reasoning.
The handbook plan wins these comparisons more often than the baseline plan: 38.3% against 28.3% on Codex, and 45.6% against 26.7% on Terminus-2; the remaining comparisons are ties. All three judges agree on the direction. Planner token use falls at the same time, by 12.7% on Codex and 8.6% on Terminus-2. The gain therefore does not come from a larger token budget.
The second result matters for cost planning. The authors compare the mid-tier planner's predicted edit sites against reference plans from Opus 4.8 and GPT-5.5. With the handbook, all 24 recall, precision, and F1 comparisons improve (F1 combines precision and recall into one score), with F1 gains of 5.0 to 18.8 points. Complete misses — plans with zero overlap with the reference — fall by up to 25.9 points. A weaker model with a good map located edit sites at close to frontier-model level. The gains hold across all request types and difficulty labels, and they are largest for scattered sites, rarely executed paths, and cross-module interactions.
The study has clear limits. The study scores plans, not executed patches: no result shows that the planned edits compile, pass tests, or resolve the request. Plan quality is judged by LLMs, and the localization references are themselves LLM-generated plans, not human ground truth. The benchmark is 60 requests across two harnesses. On Codex the per-dimension win-rate gains are small (1 to 3 points), and in both arms most comparisons end in ties. The paper counts planner tokens but does not report the token cost of building and resynchronizing the handbook itself, so the net saving is unknown.
What you can do with it
The pattern transfers to any codebase where one behavior crosses many modules, which includes most production services, not only agent harnesses. Generate a behavior-indexed document from static analysis plus LLM structuring. Link every entry to a source locator. Validate the locators in CI, freeze the entries that no longer resolve, and trigger an incremental resync from every merged diff.
The freeze rule is the transferable design idea. Documentation that an agent reads is a cache of the repository, and a cache needs an invalidation rule. Most agent memory files (a CLAUDE.md, an architecture doc) have none, so they drift from the code and the agent trusts stale claims. Here, every claim carries a locator, a locator that fails to resolve invalidates the claim, and an invalid claim is excluded instead of silently trusted. That rule costs little to adopt and works without the rest of the system.
The results also suggest a cost strategy for agent fleets: spend once on a good map, then run a cheaper planner against it, instead of paying for a frontier model to rediscover the codebase on every request. The construction pipeline and evaluation materials are linked from the project page, and the paper is CC BY 4.0.