HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Research Question and Approach

As large language model (LLM) agents are increasingly deployed in complex environments, the design of the 'harness'—the software layer orchestrating prompts, tools, and memory—has become a primary bottleneck for performance. While automatic harness evolution (AHE) allows agents to iteratively improve their own scaffolds, existing methods often suffer from overfitting to specific search tasks, misinterpreting failure signals, and cross-component interference. HarnessCompass addresses these issues by introducing a disciplined evolution framework built on three core principles: constrained evolution, proactive feedback, and component-wise optimization.

The HarnessCompass Framework

• Constrained Evolution: To prevent the meta-agent from hardcoding task-specific shortcuts, the framework enforces a global generalization gate. This gate restricts modifications to task-agnostic changes, ensuring that evolved components are reusable across diverse, unseen tasks.
• Proactive Feedback: Rather than relying solely on external trajectory logs, HarnessCompass prompts the agent to provide first-person feedback on its own experience. This feedback is then 'grounded' against the actual execution trace to filter out spurious claims or misattributions, providing the meta-agent with richer, verifiable evidence of where the harness fails.
• Component-wise Optimization: Instead of modifying the entire harness at once, the framework separates structural components (e.g., tools, middleware) and guidance components (e.g., system prompts, memory) into parallel optimization tracks. It then uses an R3 (Revision, Recombination, and Refinement) merging process to integrate beneficial changes from both tracks while discarding conflicting or redundant edits.

By replacing unrestricted evolution with a structured, disciplined approach, HarnessCompass significantly improves the efficiency and reliability of agent development. In experiments on the SWE-bench Verified benchmark using GPT-5.4, the framework increased Pass@1 from 54% to 66% in just five iterations. Crucially, the evolved harnesses demonstrated superior transferability to held-out tasks and other models, proving that the framework successfully learns generalizable design principles rather than merely memorizing the training set.