On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.
Alex: Welcome to another episode of ResearchPod. Sam, what are we looking at today?
Sam: We're examining a paper about training smaller language models to reason through complex mathematical proofs — by learning from much larger models that are already very good at it.
Alex: So it's about passing down advanced reasoning skills from a bigger, more capable model to a smaller one?
Sam: Exactly. And the core challenge is surprisingly tricky. These models don't all process text the same way. Every language model chops text into small chunks before working with it — a bit like how you might break a sentence into individual words, or even syllables. Different models use different chopping rules, and those chunks are called tokens. When the teacher and student use different tokenization systems, directly transferring knowledge tends to cause serious problems.
Alex: What kind of problems?
Sam: Training becomes unstable. The student model starts producing responses that grow longer and longer without stopping — what the paper calls "length explosions." And sometimes the output just gets cut off mid-thought because it ran out of allowed space. The whole process derails.
Alex: So how do the authors fix it?
Sam: Their key insight is to stop trying to match the teacher's and student's internal chunks directly — because those chunks don't line up. Instead, they compare only the parts of the text where both systems happen to agree on the exact same span of characters. Think of it like two people highlighting the same sentence in a book using different colored pens. You only trust the overlap. The paper calls this cross-tokenizer shared-span alignment.
Alex: So you're not forcing a translation between two incompatible systems — you're just working with the common ground.
Sam: Right. And that alone solves the vocabulary mismatch. But it doesn't fully solve the length problem.
Alex: Why not?
Sam: Because the student model can still drift. Imagine you're learning to write essays by reading a very long-winded teacher's work. You might start producing essays that are three times longer than they need to be, just because that's what you've been exposed to. The model needs a way to stay grounded.
Sam: The authors handle this with two tools. First, they mask what are called "advantage signals" on special termination tokens — the markers that tell the model it's finished. If the model gets rewarded for copying the teacher's long output, it might accidentally treat those stop signs like regular words and keep writing past them. Masking means the training system simply ignores those endpoints, so the model doesn't learn bad habits around them.
Alex: That sounds like it patches the very end of the output. But what about rambling in the middle?
Sam: Exactly the right question. That's where the second tool comes in. The authors add a mathematical penalty that measures how far the student's current behavior has drifted from where it started. If you imagine the model's original writing style as a home base, this penalty is like a tether — the further you wander, the harder it pulls you back. Statisticians call this measure KL divergence. It's essentially a score of how much one set of probabilities has strayed from a reference.
Alex: So it keeps the student learning new reasoning skills without completely losing its original sense of how to write coherently.
Sam: Precisely. And the paper found that getting the strength of that tether right really matters. Too loose, and the model still drifts. Too tight, and it can't learn anything new. The authors found a moderate setting that kept training stable and brought the truncation rate — those mid-thought cutoffs — down to nearly zero.
Alex: And did that actually translate into better performance on hard mathematical tasks?
Sam: Meaningfully so. On a challenging proof benchmark called ProofBench, one of the tested models roughly doubled its score after distillation with this method — and ended up outperforming several larger, well-known systems.
Alex: That's a significant jump. Did the length of the training examples matter too?
Sam: It did. When the researchers increased how long the training sequences were allowed to be — giving the student more room to see complete, multi-step arguments — performance on complex proof tasks improved consistently. If you're trying to learn how to follow a long logical argument, you need to actually see the whole thing, not just a fragment.
Alex: And is this method tied to one specific family of models, or does it generalise?
Sam: The authors tested it across different model families, including a switch to a model called DeepSeek-V4-Flash, and the approach held up. That's actually one of the more meaningful findings — it suggests the technique isn't dependent on a particular architecture or training lineage.
Alex: Though I imagine there are still limits.
Sam: There are. The authors are candid about this. When the tokenizer differences between teacher and student are very large, the shared-span alignment has less to work with, and the gains are more modest. It's not a universal fix — it's a meaningful step forward within a defined range of conditions.
Alex: So the practical upshot is that smaller, open-source models can inherit sophisticated reasoning abilities without needing enormous compute budgets — as long as the alignment between teacher and student is handled carefully.
Sam: That's the core contribution. It offers a more accessible pathway for smaller models to develop complex reasoning behaviours — and it does so in a way that's stable, scalable, and doesn't require the teacher and student to share the same internal language.
Alex: Thanks for walking us through it, Sam. And thanks to everyone listening to ResearchPod.