AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Large Language Models often exhibit jagged intelligence, performing well on some tasks while failing on others that appear closely related. This lack of reliability is particularly pronounced in autonomous, multi-step, and long-horizon tasks where local errors accumulate over extended interactions. Although external harnesses can significantly improve agent robustness, designing and tuning these harnesses is a manual, expensive, and non-trivial process that requires searching across large spaces of prompts, tool configurations, and control logic. The authors investigate whether harness optimization can be automated to adapt agents efficiently to new underlying models or domains.

AutoSaddler formulates harness optimization as an offline learning problem using a mini-batch training paradigm. The framework operates through an iterative loop consisting of an exploration phase and a refinement phase. At each iteration, the current harness is evaluated on a training mini-batch. Failed execution traces are then analyzed in a Diagnosis-Patch Session, which uses an evidence-grounded approach and structural constraints to treat the harness as code. The resulting updates are verified on the mini-batch and evaluated on a development set to assess generalization. A Reflection Session categorizes the outcomes of each update and records lessons into a directed acyclic graph called EvoDAG, which the Evolution Session uses to guide the synthesis of subsequent candidate harnesses.

Evaluated on three challenging agent benchmarks—GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0—AutoSaddler substantially improves long-horizon performance over baseline harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively, and outperforming stronger automated baselines. Ablation studies and qualitative evaluations indicate that successful harness optimization relies on three primary ingredients: deep debugging rather than shallow reflection, targeted modifications instead of unconstrained code editing, and generalization-aware selection rather than narrow, trajectory-specific repair.