Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

In multi-turn agentic settings, standard On-Policy Self-Distillation (OPSD) often fails because the "self-teacher" model—which provides dense, logit-level supervision—suffers from exposure bias. This leads to reasoning route convergence, where the model loses its ability to explore effectively and fails to maintain clear optimization directions over long-horizon tasks. The authors investigate how to refine this distillation process to improve training stability and generalization.

The authors introduce Contrastive Reinforced Policy Optimization (CRPO), which treats self-distillation as a contrastive learning problem. Instead of blindly imitating the teacher at every position, CRPO uses predictive entropy to calculate the uncertainty difference between the student and the privileged self-teacher. It categorizes positions into "positive" (where the teacher engages in reflective exploration) and "negative" (where the teacher likely suffers from exposure bias). By applying a group-wise contrastive loss (InfoNCE), the model is encouraged to align with the teacher at positive positions while actively penalizing imitation at negative ones.

CRPO consistently outperforms standard OPSD and existing reinforcement learning baselines (such as GRPO and ARPO) across 13 challenging reasoning and deep-search benchmarks. The authors demonstrate that by using entropy-based gating, the model preserves more reliable optimization signals, leading to better performance in complex, multi-turn interactions. Theoretical analysis shows that the CRPO gradient acts as a per-token policy gradient with a soft gate, effectively balancing exploration and exploitation without requiring additional computational overhead.

This work addresses a critical bottleneck in agentic LLM training: the tendency for self-distillation to cause models to "collapse" into narrow, over-confident reasoning paths. By providing a principled way to filter out unreliable supervision signals, CRPO offers a more robust framework for training agents that must interact with external tools and environments over multiple turns, where sparse outcome-level rewards are often insufficient.