H2SD: Hybrid Hindsight Self-Distillation

Large language models (LLMs) often struggle with complex reasoning tasks due to sparse supervision from scalar outcome rewards. While on-policy distillation (OPD) and on-policy self-distillation (OPSD) offer denser token-level supervision, they often face trade-offs between training stability, information leakage, and the need for external teacher models. This paper investigates how to optimize the use of teacher signals to improve reasoning performance without sacrificing stability.

The authors introduce Hybrid Hindsight Self-Distillation (H2SD), a framework that dynamically selects the supervision strategy based on the correctness of the student-generated trajectory. For successful trajectories, the model uses a rephrasing-conditioned teacher to modulate update magnitudes, refining credit assignment without altering the reward-determined direction. For failed trajectories, the model uses a hint-conditioned teacher to provide explicit corrective guidance by minimizing the reverse KL divergence between the student and the teacher distributions. This hybrid approach aims to combine the stability of magnitude-based modulation with the corrective power of distribution distillation.

Experiments on logical reasoning benchmarks—including Sudoku, Calcudoku, and Arrow Maze—demonstrate that H2SD consistently outperforms representative baselines such as GRPO, OPSD, and RLSD. By tailoring the distillation strategy to the outcome of the reasoning process, the model achieves more stable optimization and higher accuracy, effectively leveraging privileged information (hints) to guide the student's reasoning path.

H2SD provides a robust mechanism for improving LLM reasoning by addressing the limitations of scalar reward-based reinforcement learning. By demonstrating that different supervision strategies are optimal for different outcomes, the paper offers a practical path for training more capable reasoning agents while mitigating common issues like training instability and information leakage.