Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance | Zhuowen Han et al. | ResearchPod

Transcript: Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study that explores how to make artificial intelligence models better at reasoning — specifically in fields like mathematics and computer programming.

Sam: So the paper is basically asking how we can use a "teacher" model to help a "student" model learn more effectively when it gets stuck?

Alex: Exactly. The central claim is that we shouldn't force a student model to copy a teacher all the time. Instead, guidance should only kick in when the student is completely failing to solve a problem on its own.

Sam: And the core problem is that if you teach the model too much, or at the wrong time, it stops thinking for itself and just becomes a mirror of the teacher.

Alex: That's the core challenge. The authors describe a situation they call "negative zero-variance" loops. Think of it like this: imagine a student who gets every single practice problem wrong, every single time. Because they never see a correct answer from their own work, they have no idea what they're doing differently from someone who gets it right. There's no signal telling them how to improve — just a wall of failure.

Sam: That's a tough spot to be in. And I imagine a computer model faces the exact same problem?

Alex: Precisely. The standard way to train these models is through something called reinforcement learning. The model tries a few different approaches to a problem, and when it gets one right, it gets a reward — like points in a game. Over time, it learns to do more of what earns rewards.

Sam: But if the model is so bad at the task that it never gets a reward, the system has nothing to work with?

Alex: Right. The signal that tells the model how to improve — the mathematical nudge in the right direction — effectively disappears. Because all the attempts are equally wrong, there's no difference between them to learn from.

Sam: So how does this new method — RSTG — actually fix that?

Alex: RSTG stands for "Recovering Learning Signals via Adaptive Teacher Guidance." The key word is "adaptive." Instead of applying teacher guidance to every problem the student sees, it only activates on the specific problems where the student is completely stuck — where every single attempt fails, but the teacher knows the correct path.

Sam: So it ignores the problems the student can already handle, and only steps in when they're genuinely lost?

Alex: Exactly. And crucially, it doesn't just hand the student the answer. That would create the dependency problem we mentioned. Instead, it uses a technique the authors call "on-policy distillation." Think of it like a dance instructor. Rather than demonstrating the whole routine from the top, the instructor watches you move and gives precise feedback only on the specific steps where you're stumbling — "your left foot should go here at this moment, not there."

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

Zhuowen Han, Jinwei Xiao, Zhengxi Lu, Renren Jin, Zhiyuan Yao, Yuxin Liu, Hongyan Hao, Yueqing Sun, Yu Yang, Qi Gu, Xunliang Cai, Deyi Xiong

Aug 1, 2026·5:16·arxiv·Original Paper

Reinforcement Learning with Verifiable Rewards (RLVR)Group Relative Policy Optimization (GRPO)On-Policy Distillation (OPD)Negative Zero-Variance PromptsRSTG (Recovering Learning Signals via Adaptive Teacher Guidance)

Listen

5 min

Briefing

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance: Research Paper Podcast

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

Listen