Weak-to-Strong Generalization via Direct On-Policy Distillation | Shiyuan Feng et al. | ResearchPod

Transcript: Weak-to-Strong Generalization via Direct On-Policy Distillation

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper that tackles a major bottleneck in AI: the high cost of teaching large language models how to reason.

Sam: Right. So here's the situation. When researchers want an AI to get better at reasoning—working through a math problem step by step, for instance—they use a process called reinforcement learning. The basic idea is that the model tries something, gets feedback on whether it was right or wrong, and gradually adjusts. The version used for reasoning is called reinforcement learning with verifiable rewards, or RLVR for short. The problem is that running this process on a large, powerful model is extraordinarily expensive. We're talking days of computation on high-end hardware. And you have to repeat that cost every time you want to train a new, bigger model.

Alex: So this paper is asking: do we actually have to pay that cost every single time?

Sam: That's exactly the question. The authors propose a method they call Direct On-Policy Distillation. The core idea is to run the expensive learning process on a small, cheap model instead. Then, rather than training the large model from scratch, you transfer what the small model learned across to it.

Alex: But that sounds like it could go wrong pretty quickly. If the large model is already more capable, why would copying from a weaker one help?

Sam: That's the key insight the paper builds on. If you simply copy the small model's final answers, you do drag the large model down. You're essentially telling a more capable student to mimic someone who knows less. That's counterproductive.

Alex: So what do they do instead?

Sam: They look at the change the small model went through, rather than where it ended up. Think of it like watching a student practice a skill. Instead of copying their final attempt, you study how their technique shifted from their first try to their last. You're capturing the direction of improvement, not the endpoint. Technically, they compare the small model's behavior before training and after training, and calculate what they call a "policy shift"—the difference between those two states.

Alex: Oh—so the large model isn't learning what the small model knows. It's learning which direction to move in.

Sam: Exactly. That shift acts as a kind of signal—the paper calls it an implicit reward—that tells the large model which ways of thinking tend to lead to better answers. The large model can then follow that signal without ever having to run the slow trial-and-error process itself.

Alex: That's a meaningful distinction. And does it actually hold up in practice?

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Jul 6, 2026·4:28·arxiv·Original Paper

Direct On-Policy Distillation (Direct-OPD)Reinforcement Learning with Verifiable Rewards (RLVR)Weak-to-Strong GeneralizationPolicy Shift TransferDirect-OPDPolicy Shift

Listen

4 min

Briefing

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

Weak-to-Strong Generalization via Direct On-Policy Distillation: Research Paper Podcast

Weak-to-Strong Generalization via Direct On-Policy Distillation

Listen