Haoyang Yan, Min-Le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, Shuyue Hu
5 min
Abstract
This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/
Alex: The ablations are informative here. When you remove the evidence feedback — keeping the iterative structure but stripping the validated knowledge transfer — performance drops substantially. When you remove the warm-start mechanism, which initialises each cycle from the prior artifact state rather than from scratch, you see a similar degradation. That pattern suggests the state transfer is the active ingredient, not simply the additional compute from running more iterations. More cycles without structured handoff doesn't replicate the gains.
Sam: That's an important distinction. Though I'd imagine a careful referee would push back on resource cost. More iterations, more structured artifacts — the token budget must grow considerably.
Alex: That's a real constraint. The framework is explicitly trading compute for reliability, and token usage scales with iteration count. The authors acknowledge this, and it's the most practical limitation for real-world deployment. You're buying robustness with compute, which means HoH isn't a drop-in replacement for single-pass agents in cost-sensitive settings.
Sam: And there's a deeper structural risk, isn't there? The whole framework depends on the QA tester accurately diagnosing the artifact state. If that diagnostic loop is wrong—
Alex: —the error propagates forward. If the QA agent mislabels a failure, that bad signal gets written into the evidence state, and the planner inherits it. You can end up with the agent locked onto a suboptimal path, confidently iterating away from the correct solution because its own validated knowledge is corrupted. The framework is only as reliable as the independence and accuracy of that evaluation loop. That's the most significant limitation the paper doesn't fully resolve.
Sam: So the QA tester is load-bearing in a way that isn't always visible from the headline results.
Alex: Precisely. And it points to where the next generation of this work needs to go — either more robust QA agents, or mechanisms for detecting and correcting corrupted evidence states mid-deployment. The current framework assumes the diagnostic loop is trustworthy. That assumption holds well enough in the reported experiments, but it's where a real-world deployment is most likely to diverge from the benchmark results.
Sam: So the honest read is: meaningful architectural contribution, but with a ceiling determined by QA quality.
Alex: That's right. The iterative, evidence-grounded structure is a more principled approach than single-pass generation, and the results support that. But autonomous software engineering at scale will require either much stronger diagnostic agents, or a way to bring human judgment back into the loop at the right moments — without collapsing back into the supervised workflows this framework is trying to move beyond. Thanks for listening to ResearchPod.