On-Policy Self-Distillation without Any Supervision

Existing on-policy distillation methods for large language models (LLMs) typically rely on external supervision, such as ground-truth solutions, environmental feedback, or guidance from larger models. This dependence limits their scalability and applicability in scenarios where high-quality supervision is unavailable or costly. The authors investigate whether a model can perform genuine self-distillation by constructing its own privileged context entirely from its own internal consistency.

The authors propose Unsupervised On-Policy Self-Distillation (U-OPSD). For a given unlabeled problem, the model generates multiple independent rollouts. It then uses a majority-vote mechanism to identify a pseudo-solution from the agreeing rollouts. Once a consensus is reached (based on a self-consistency threshold), the model treats the shortest agreeing rollout as a teacher reference. It then performs token-level distillation by conditioning the teacher on this pseudo-solution and applying it to the prefixes of the model's own incorrect (disagreeing) completions. This process allows the model to learn from its own mistakes by aligning its behavior with its own consensus.

U-OPSD consistently improves base model performance across diverse mathematical reasoning benchmarks, including AIME24, AIME25, HMMT25, MATH500, and AMC23. In non-thinking mode, U-OPSD achieved significant gains over base models (8.5% to 10.7% at 4B and 8B scales) and outperformed supervised methods like OPSD and GRPO. In thinking mode, U-OPSD remained competitive with or superior to supervised baselines. The results demonstrate that consensus-based self-distillation is a highly effective, label-free alternative to traditional supervised fine-tuning or reinforcement learning with external rewards.

This work demonstrates that LLMs can effectively self-evolve and improve their reasoning performance without any external supervision. By removing the need for gold-standard datasets, U-OPSD provides a scalable path for post-training models in domains where human-labeled data is scarce or expensive, effectively bridging the gap between unsupervised self-training and supervised distillation.