PROBE-AND-REFINE TUNING OF REPOSITORY GUIDANCE FOR CODING AGENTS

Coding agents often struggle to navigate complex repositories because they lack the implicit operational knowledge—such as subsystem boundaries, debugging workflows, and test-running conventions—that experienced human developers possess. While engineers often create AGENTS.md files to provide this context, prior research has been divided on whether these files actually improve agent performance. This paper investigates whether the effectiveness of repository guidance depends on how it is produced, rather than the guidance itself, and introduces a systematic method for generating high-quality instructions.

The authors introduce "probe-and-refine tuning," a lightweight, iterative procedure that generates repository-specific guidance. The process starts with a static knowledge base derived from tree-sitter parsing and generic best practices. It then enters a loop where the system generates synthetic bug-fix tasks (probes), attempts to solve them using the current guidance, and uses the resulting failures to diagnose and patch the guidance file. This process uses only single-shot LLM calls, avoiding complex agent loops or tool use during the tuning phase. The authors evaluate this method on the SWE-bench Verified benchmark using the Qwen3.5-35B-A3B model, comparing it against unguided and static-guidance baselines.

Probe-and-refine tuning significantly improves agent performance, achieving a 33.0% resolve rate compared to 28.3% for static guidance and 25.5% for no guidance. The analysis reveals that this improvement is driven by increased coverage—the agent is more likely to identify the correct file to modify—rather than an increase in the precision of the patches themselves. The authors also find that refined guidance allows agents to utilize larger step budgets more productively. However, a cross-model experiment demonstrates that the tuning loop is sensitive to model capability; if the model cannot generate sufficiently diagnostic feedback during the refinement process, the guidance fails to improve performance.