Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study that explores how to make artificial intelligence models better at reasoning — specifically in fields like mathematics and computer programming.
Sam: So the paper is basically asking how we can use a "teacher" model to help a "student" model learn more effectively when it gets stuck?
Alex: Exactly. The central claim is that we shouldn't force a student model to copy a teacher all the time. Instead, guidance should only kick in when the student is completely failing to solve a problem on its own.
Sam: And the core problem is that if you teach the model too much, or at the wrong time, it stops thinking for itself and just becomes a mirror of the teacher.
Alex: That's the core challenge. The authors describe a situation they call "negative zero-variance" loops. Think of it like this: imagine a student who gets every single practice problem wrong, every single time. Because they never see a correct answer from their own work, they have no idea what they're doing differently from someone who gets it right. There's no signal telling them how to improve — just a wall of failure.
Sam: That's a tough spot to be in. And I imagine a computer model faces the exact same problem?
Alex: Precisely. The standard way to train these models is through something called reinforcement learning. The model tries a few different approaches to a problem, and when it gets one right, it gets a reward — like points in a game. Over time, it learns to do more of what earns rewards.
Sam: But if the model is so bad at the task that it never gets a reward, the system has nothing to work with?
Alex: Right. The signal that tells the model how to improve — the mathematical nudge in the right direction — effectively disappears. Because all the attempts are equally wrong, there's no difference between them to learn from.
Sam: So how does this new method — RSTG — actually fix that?
Alex: RSTG stands for "Recovering Learning Signals via Adaptive Teacher Guidance." The key word is "adaptive." Instead of applying teacher guidance to every problem the student sees, it only activates on the specific problems where the student is completely stuck — where every single attempt fails, but the teacher knows the correct path.
Sam: So it ignores the problems the student can already handle, and only steps in when they're genuinely lost?
Alex: Exactly. And crucially, it doesn't just hand the student the answer. That would create the dependency problem we mentioned. Instead, it uses a technique the authors call "on-policy distillation." Think of it like a dance instructor. Rather than demonstrating the whole routine from the top, the instructor watches you move and gives precise feedback only on the specific steps where you're stumbling — "your left foot should go here at this moment, not there."
Sam: Oh, so it's not "here is the answer." It's "here is how you should have reasoned through this specific step."
Alex: That's a good way to put it. The student model learns by aligning its own step-by-step choices with the teacher's, but only at the moments where the student's own logic is breaking down. It's targeted, not wholesale.
Sam: You mentioned the method also uses something called "auxiliary supervised fine-tuning." How does that fit in?
Alex: That's the second tool in the kit. When the student is completely lost, the system also provides a fully worked "gold standard" example from the teacher — a solved problem to study. But only after the student has already tried and failed on its own. It injects a positive signal where the reinforcement learning system was previously giving the model nothing at all.
Sam: And did it actually make the models better?
Alex: The paper suggests it did. Across several benchmarks in math and code, they saw a meaningful improvement in accuracy compared to the standard approach of combining these methods. And the gains held up across different pairings of teacher and student models.
Sam: So the key insight is that the timing of the feedback matters just as much as the feedback itself.
Alex: That's the primary takeaway. It's about efficiency — using the teacher's knowledge only when it's truly needed to bridge a gap, rather than flooding the student with guidance it doesn't need.
Sam: Are there any real downsides?
Alex: The authors are careful about this. The method relies heavily on having a teacher model that is significantly more capable than the student. If the teacher isn't genuinely better at the task, that surgical guidance might just lead the student down the wrong path. And the research is currently validated on reasoning-heavy domains — math and code — so how well it transfers to more open-ended tasks is still an open question.
Sam: So it's a specialized tool for a specific kind of learning problem, not a fix for everything.
Alex: Precisely. It's a meaningful step in how we structure these training loops. The authors also suggest it could point toward more autonomous systems — where a model might eventually identify its own knowledge gaps and request teacher input, rather than needing a human to design when that happens.
Sam: That would be a significant shift. Moving from a fixed training process to something closer to a real-time tutoring relationship.
Alex: That's the longer-term possibility the paper gestures toward, though we're still in the early stages of making that practical.
Sam: It's a clear example of how a targeted adjustment to a training process — not a wholesale redesign — can recover something that was otherwise being lost.
Alex: And that's perhaps the broader lesson. In machine learning, being more selective about how and when you use the information you already have can matter just as much as having more data in the first place. Thanks for listening to ResearchPod.