Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, Leoweiliang
6 min
Modern AI agents rely on complex harnesses—the software infrastructure that manages prompts, state, tool invocation, and execution flow. As these systems evolve, developers and coding agents must modify the harness to adapt to new requirements. However, modification requests are typically described in terms of behavior, while codebases are organized by files and modules. This paper addresses the 'behavior localization' bottleneck: the difficulty of identifying all scattered code locations that implement a specific system behavior.
The authors introduce the Harness Handbook, a structured, behavior-centric representation of a harness codebase. The handbook is constructed through a three-phase pipeline:
To utilize this, the authors developed Behavior-Guided Progressive Disclosure (BGPD). This workflow guides agents from high-level behavioral requests down to specific code snippets, verifying candidate locations against the current repository state before planning edits. The system also includes an automated resynchronization mechanism that updates the handbook whenever the underlying code is modified.
Evaluation on diverse modification requests across two open-source agent harnesses demonstrates that Handbook-Assisted planning significantly improves both behavior localization and the quality of edit plans. By explicitly linking behavior to code, the system reduces the number of planner tokens required to navigate the repository. The benefits are most pronounced for complex changes, such as those involving scattered implementation sites, rarely executed code paths, and cross-module interactions.
As agentic systems grow in complexity, the 'harness' becomes a critical, yet often opaque, layer of the stack. By shifting the focus from file-based navigation to behavior-based navigation, the Harness Handbook reduces the cognitive load on developers and improves the reliability of automated coding agents. This approach suggests that the future of software maintenance for agentic systems lies in maintaining explicit, machine-readable mappings between runtime behavior and static source code.
The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing ease inspection, but still leave this behavior-to-code mapping to be recovered by hand. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its corresponding source. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source. On diverse modification requests from two open-source harnesses, Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains on scattered sites, rarely executed paths, and cross-module interactions. Evolving complex agentic systems thus depends not only on generating edits, but also on determining where those edits should be made.
Alex: It's all about the index, not just the raw processing power.
Sam: That's a clean way to put it. The deeper implication is that maintaining complex software depends as much on knowing where to make a change as it does on having the skill to write the change itself.
Alex: That raises a practical question, though. Software is never finished—teams are constantly adding features and fixing bugs. How does the handbook stay accurate as the code changes underneath it?
Sam: That's a critical concern, and the paper addresses it directly. They call the solution "Resynchronization." Whenever a change is made to the codebase, an automated process re-reads the affected code and updates the handbook to match. If a function moves to a different file, the handbook is corrected. If a function is deleted entirely, the handbook marks that entry as invalid so the agent isn't sent on a dead-end search.
Alex: So it's self-correcting. Like a map app that updates when a road closes.
Sam: That's the intent, yes. The key insight is that an outdated map is potentially worse than no map at all—it actively misleads the agent. So keeping the handbook synchronized with the live code isn't a nice-to-have; it's essential to the whole approach working.
Alex: That makes sense. But how do you actually measure whether the agent's plan is any good? Finding the right files is one thing—knowing whether you've found all the right files is another.
Sam: The researchers thought carefully about this. Before the agent attempts a task, the team creates what they call an "Answer Key"—a pre-defined list of every file, function, and variable that must be touched to correctly complete the change. This is done independently, before seeing what the AI does, to avoid any bias.
Alex: So it's like a teacher writing the answer sheet before handing out the test.
Sam: Exactly. And the grading looks at three things. First: did the agent find all the necessary locations? Second: did it stay focused, or did it wander into unrelated files? Third: is the agent's reasoning actually grounded in the code, or is it just making plausible-sounding guesses? They call this evaluation framework the "Terminus-2 judge."
Alex: So a perfect score means finding every correct spot, nothing extra, with solid reasoning. That does sound like a high bar.
Sam: It is. And that's partly the point. The framework is designed to reveal exactly where planning breaks down—whether an agent misses a scattered dependency, or confidently edits the wrong file because it looked superficially relevant.
Alex: And the handbook is meant to prevent those misses by giving the agent that structured path through the hierarchy.
Sam: That's the core claim, yes. Though the paper is careful to note that the handbook is only as reliable as the automated analysis that builds it. If that analysis fails to correctly parse an unusual code structure, or misclassifies a function, the handbook's accuracy suffers—and at that point, a human would need to step in and correct it.
Alex: So the map is only as good as the surveyor.
Sam: A fair summary. But the researchers suggest that as these tools mature, a handbook like this could become a kind of shared memory across multiple agents—something that accumulates knowledge about a codebase over time, allowing agents to build on each other's work rather than starting from scratch every time.
Alex: That's a meaningful shift in how we think about AI and software. Not just a tool that writes code on demand, but something that genuinely understands the architecture it's working within.
Sam: And that understanding, the paper argues, is what separates an agent that can make a small, targeted change from one that inadvertently breaks something three modules away. The Harness Handbook is one attempt to give agents that structural awareness.
Alex: A well-drawn map, it turns out, might matter as much as the skill of the person reading it. Thanks for listening to ResearchPod.