SAMPLE-EFFICIENT LEARNING FROM AGENT EXPERIENCE

How can agents internalize the benefits of in-context learning (ICL) into their own model weights without incurring the high costs of further environment interaction? While ICL allows agents to adapt to tasks using interaction histories, these gains are lost once the history is removed from the context. This paper introduces Experience Distillation to consolidate these gains into the model's parameters.

The authors propose a framework that treats experience distillation as a trajectory-level context distillation problem. To avoid the need for additional environment rollouts or unreliable world models, they implement a one-step branched rollout strategy. In this approach, a teacher model, conditioned on the collected experience, generates a single decision at various points along the recorded trajectory. These decisions are then used to train a student model via supervised learning. The authors further optimize this process using 'branch packing,' which combines multiple teacher-generated decisions from a single trajectory into a more efficient training sequence.

Experiments across 749 software-engineering tasks and six text-adventure games demonstrate that Experience Distillation is highly effective. On software-engineering tasks, the distilled models retained 64.8% of the performance gains observed in ICL, whereas standard supervised fine-tuning (SFT) recovered only 3.8%. Furthermore, the approach matches or exceeds the performance of classical reinforcement learning baselines (PPO and GRPO) while requiring at least 9.6 times fewer environment samples. The results suggest that one-step distillation is superior to multi-step model-based rollouts, as it avoids the compounding errors inherent in long-horizon synthetic trajectories.

This research provides a practical pathway for creating sample-efficient agents in domains where environment interaction is expensive, such as scientific discovery, legal analysis, or complex software engineering. By successfully distilling trial-and-error experience into model weights, the authors demonstrate that agents can achieve high performance without the massive sample requirements typically associated with reinforcement learning.