Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, Yu Cheng
5 min
Abstract
On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.
Alex: That sounds like it patches the very end of the output. But what about rambling in the middle?
Sam: Exactly the right question. That's where the second tool comes in. The authors add a mathematical penalty that measures how far the student's current behavior has drifted from where it started. If you imagine the model's original writing style as a home base, this penalty is like a tether — the further you wander, the harder it pulls you back. Statisticians call this measure KL divergence. It's essentially a score of how much one set of probabilities has strayed from a reference.
Alex: So it keeps the student learning new reasoning skills without completely losing its original sense of how to write coherently.
Sam: Precisely. And the paper found that getting the strength of that tether right really matters. Too loose, and the model still drifts. Too tight, and it can't learn anything new. The authors found a moderate setting that kept training stable and brought the truncation rate — those mid-thought cutoffs — down to nearly zero.
Alex: And did that actually translate into better performance on hard mathematical tasks?
Sam: Meaningfully so. On a challenging proof benchmark called ProofBench, one of the tested models roughly doubled its score after distillation with this method — and ended up outperforming several larger, well-known systems.
Alex: That's a significant jump. Did the length of the training examples matter too?
Sam: It did. When the researchers increased how long the training sequences were allowed to be — giving the student more room to see complete, multi-step arguments — performance on complex proof tasks improved consistently. If you're trying to learn how to follow a long logical argument, you need to actually see the whole thing, not just a fragment.
Alex: And is this method tied to one specific family of models, or does it generalise?
Sam: The authors tested it across different model families, including a switch to a model called DeepSeek-V4-Flash, and the approach held up. That's actually one of the more meaningful findings — it suggests the technique isn't dependent on a particular architecture or training lineage.
Alex: Though I imagine there are still limits.
Sam: There are. The authors are candid about this. When the tokenizer differences between teacher and student are very large, the shared-span alignment has less to work with, and the gains are more modest. It's not a universal fix — it's a meaningful step forward within a defined range of conditions.
Alex: So the practical upshot is that smaller, open-source models can inherit sophisticated reasoning abilities without needing enormous compute budgets — as long as the alignment between teacher and student is handled carefully.
Sam: That's the core contribution. It offers a more accessible pathway for smaller models to develop complex reasoning behaviours — and it does so in a way that's stable, scalable, and doesn't require the teacher and student to share the same internal language.
Alex: Thanks for walking us through it, Sam. And thanks to everyone listening to ResearchPod.