Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

Most research on agent-harness optimization—improving prompts, tools, and control code—focuses on one-shot performance against a static benchmark. However, real-world agents are rarely static; they are updated recursively as new tasks arise or failures are discovered. This study investigates whether optimization gains actually compound—that is, whether an agent can be optimized on new tasks without losing the performance gains achieved in previous rounds.

The authors introduce a two-phase continual-learning evaluation using Terminal-Bench 2.0. In Phase 1, agents are optimized on an initial task set. In Phase 2, the agents are further optimized on an expanded set that includes new, unseen tasks. This protocol allows the researchers to distinguish between three critical properties: static optimization strength, transferability to unseen tasks, and the ability to continue improving without regressing on previously solved tasks.

The study compares three approaches: GEPA (evolutionary prompt search), Meta Harness (outer-loop code editing), and RELAI-VCL (regression-aware continual learning). While all three methods improved upon the baseline in the initial static phase, their performance diverged significantly during the second phase:

• GEPA overfitted to the initial tasks, leading to negative transfer where performance on the expanded set dropped below the unoptimized baseline.
• Meta Harness demonstrated good transfer but stalled during the second optimization phase, failing to make further progress.
• RELAI-VCL was the only method to show both positive transfer and continued improvement, achieving the highest lifelong average pass rate (76.4%).

The findings suggest that current optimization methods often conflate static performance with the ability to learn continuously. The success of RELAI-VCL highlights that regression control—the explicit rejection of candidate edits that harm previously mastered tasks—must be an active constraint within the search loop rather than a post-hoc check. This is essential for building agents that remain robust and capable as they are updated over time.