Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

Optimizing the control code or harness around a fixed large language model can yield massive performance gains without changing model weights. Traditionally, automated harness optimization evaluates every candidate harness against a fixed, full validation task set at every iteration. This creates a severe computational bottleneck, especially for expensive long-horizon tasks, and wastes evaluation budget on tasks that become uninformative as the harness improves. This paper asks whether harness optimization can be made efficient by dynamically selecting a small, informative subset of validation tasks at each iteration without sacrificing optimization quality or causing overfitting.

To overcome the limitations of static validation sets, the authors propose Task-CoEvolve, a framework that co-evolves validation tasks alongside candidate harnesses. The method operates in two main phases. First, it uses variance-weighted sampling based on past evaluation outcomes to focus the evaluation budget on tasks near the agent's capability frontier where candidate harnesses disagree. Second, it employs sampling-aware full-set score estimation using inclusion probabilities to ensure that performance comparisons remain fair and consistent across iterations despite evaluating different subsets.

The authors evaluate Task-CoEvolve on online text classification tasks and Terminal-Bench 2.1, a benchmark for long-horizon terminal agents. On text classification, Task-CoEvolve approaches full-set search performance with an extreme evaluation budget of only 7 percent and outperforms full-set search at a 20 percent budget. On Terminal-Bench 2.1, Task-CoEvolve matches full-set search performance while reducing the number of evaluations by 80 percent and the overall search cost by up to 80 percent.