An Empirical Study of Harness Design for Coding Agents

Coding agents rely on a harness—a software layer managing planning, action interfaces, and context—to solve long-horizon software engineering tasks. While prior research has evaluated these harnesses as monolithic systems, it remains unclear how individual components contribute to performance across different models and resource constraints. This paper systematically dissects the coding harness by holding the execution loop constant while independently varying three components: planning, action space, and context management. The authors evaluate 176 experimental settings across four models (Nemotron-3 30B/120B/550B and Mistral-Medium-3.5-128B) on two benchmarks: SWE-Bench Verified and Terminal-Bench 2.1.

• Context Management: Its primary value is preventing premature task termination due to context-window overflow. As the context budget expands, the accuracy benefit of management diminishes. Among the strategies tested, a staged approach (T4)—applying rule-based elision before LLM-based summarization—achieves the best accuracy-to-cost ratio by minimizing expensive summarization calls.
• Planning: This component acts as an accuracy scaffold for weaker models, keeping them on track long enough to attempt a fix. For stronger models, planning serves as an efficiency aid, reducing redundant post-edit verification and lowering overall costs.
• Action Space: Predefined tools (e.g., read_file, edit_file) provide necessary scaffolding for models with limited bash proficiency. Conversely, bash-capable models perform better and more efficiently with a bash-only interface, as they can compose complex, multi-step operations that the structured tool set would otherwise fragment into multiple, higher-cost interactions.

This study demonstrates that there is no universally optimal coding harness. Instead, harness design is a conditional systems problem. Developers should select components based on the target model's capabilities, the specific task type (e.g., shell-centric vs. repository-wide), and the available computational budget. The findings provide a modular framework for future harness design, moving away from monolithic evaluation toward component-aware optimization.