SEED: SELF-EVOLVING ON-POLICY DISTILLATION FOR AGENTIC REINFORCEMENT LEARNING

Large language model (LLM) agents often struggle with long-horizon tasks because standard reinforcement learning (RL) provides only sparse, episode-level rewards. This creates a supervision gap where the model receives feedback on whether it succeeded but lacks guidance on which specific intermediate actions or decisions were responsible for that outcome. The authors investigate how to bridge this gap by converting completed trajectories into reusable hindsight information.

SEED (SElf-Evolving On-Policy Distillation) introduces a two-stage framework to internalize hindsight knowledge without requiring external memory or inference-time prompts.

1. Hindsight Skill SFT: The model is fine-tuned to analyze completed interaction histories and generate natural-language skills—such as reusable workflows or failure-avoidance rules—derived from those trajectories.
2. Self-Evolving On-Policy Distillation (OPD): During RL, the current policy checkpoint serves two roles: it acts in the environment to collect new trajectories and acts as an analyzer to extract hindsight skills from those same trajectories. The model then re-scores the sampled actions under both an ordinary context and a skill-augmented context. The difference in log-probabilities between these two contexts creates a dense, token-level signal that is used to update the policy alongside the standard outcome-based RL objective.

Because the actor and the analyzer share the same model parameters, the policy's decision-making and its ability to extract useful hindsight evolve in tandem throughout the training process.

SEED consistently outperforms baseline methods across diverse agentic benchmarks, including text-based and vision-based tasks. By transforming trajectory-level outcomes into dense, token-level supervision, the model achieves higher task success rates and improved sample efficiency. Furthermore, the learned policies demonstrate robust generalization to unseen scenarios, suggesting that the internalized hindsight skills capture transferable behavioral strategies rather than just memorizing specific task instances.