HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

As LLMs are increasingly integrated into complex agentic systems, their performance depends heavily on the surrounding 'harness'—the prompts, tools, and orchestration logic. This paper investigates whether frontier LLMs can autonomously optimize these harnesses to improve agent performance. The authors seek to determine if harness optimization is a measurable, discriminative capability and how much of this performance is attributable to the model itself versus the coding environment it operates within.

The authors introduce HarnessOpt-Bench, a standardized evaluation protocol for end-to-end harness optimization. The benchmark requires an optimizer (an LLM paired with a coding harness) to improve a target agent's seed harness under a fixed evaluation budget. To ensure rigorous measurement, the benchmark uses a trusted execution environment that enforces a strict boundary: the optimizer receives feedback from development and validation sets but remains blind to a held-out test partition. The study evaluates five frontier LLMs across four downstream tasks, using both a shared coding harness and each model's native harness to isolate the impact of the model from the scaffolding.

The study establishes that harness optimization is a distinct and measurable capability. The results show that the choice of optimizer model is a more significant factor in performance than the choice of coding harness. While some models demonstrate strong optimization capabilities, native harnesses do not consistently outperform a shared, standardized coding harness. Furthermore, the researchers observed that broader, system-level interventions by the LLMs were more strongly correlated with performance gains than the use of detailed, step-by-step failure-trace inspection.

By providing a reproducible, controlled environment for testing agent meta-optimization, this benchmark shifts the field away from ad-hoc evaluations. It highlights that as agentic systems become more complex, the ability to reason about and improve the harness—rather than just generating code—is becoming a critical bottleneck for AI performance. This framework allows researchers to quantify how well models can navigate the trade-offs between exploration, budget management, and diagnostic reasoning in stochastic environments.