ResearchPod Summary
On-policy distillation (OPD) and on-policy self-distillation (OPSD) are efficient methods for training LLMs on reasoning tasks by aligning student models with teacher distributions. However, these methods typically treat all student-generated outputs (SGOs) equally. This paper investigates whether certain trajectories provide more valuable learning signals than others and proposes a method to dynamically weight these trajectories without requiring expensive full-answer rollouts.
The authors first conduct controlled experiments revealing a consistent asymmetry: training on incorrect SGOs consistently outperforms training on correct ones. They hypothesize that incorrect trajectories preserve exploratory reasoning and reflection-style markers, whereas correct-only training can lead to overly confident, shorter, and less robust reasoning.
To exploit this, they introduce ReNIO (Reweighting Negative trajectory Importance). ReNIO identifies 'pivotal tokens'—local decisions where the student's probability distribution significantly deviates from the teacher's—using a prefix-computable student-to-teacher probability ratio. These ratios are aggregated into a normalized sample weight, allowing the model to prioritize informative negative trajectories during training. Because this weighting is computed using only prefix-conditioned probabilities, ReNIO maintains the computational efficiency of standard OPD, avoiding the need for full-sequence reward labeling.
ReNIO consistently improves performance across both mathematical reasoning and code generation tasks. In experiments with Qwen3 and DeepSeek-R1-Distill-Qwen models, ReNIO demonstrated significant relative gains, including up to 8.90% for Qwen3-1.7B and 10.00% for R1-Distill-Qwen-7B on mathematical benchmarks. The results confirm that emphasizing high-disagreement, likely negative trajectories provides a more effective supervision signal than uniform sampling, without increasing the training time compared to standard OPD.
This work addresses a critical bottleneck in LLM post-training: the efficient selection of training data. By demonstrating that 'failed' reasoning attempts are often more informative than successful ones, the authors provide a scalable, prefix-based mechanism to improve reasoning capabilities. This approach bridges the gap between the efficiency of distillation and the performance benefits of reinforcement learning, without the overhead of sparse, sequence-level reward signals.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.