Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen
5 min
Abstract
Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.
Sam: And I assume there's a timing element too? The teacher presumably shouldn't hover forever.
Alex: Right. The system is designed in two phases. There's a "warm-up" phase at the start, where the teacher's guidance is welcomed — the AI is new, it needs direction. Then comes an "annealing" phase, where the teacher's influence is gradually reduced, like slowly lowering the volume on a radio. The AI has learned the basics; now it needs to develop its own judgment.
Sam: So it's a bit like learning to ride a bike. Someone holds the back of the seat at first, but eventually they have to let go — otherwise you never actually learn to balance.
Alex: That's a good way to put it. By controlling both the strength of the feedback and when it's delivered, the researchers prevent the model from getting stuck in that entropy collapse. It stays curious, keeps exploring, and doesn't just become a copy of its teacher.
Sam: Does this actually lead to better results across different types of tasks?
Alex: The paper suggests it does. They tested the approach across seven different tasks in mathematics and coding, and this method consistently outperformed the standard way of combining those two training signals. The improvement wasn't about using more computing power or more data — it was about being more precise with how the AI receives its corrections.
Sam: Though I suppose there are limits. They only tested this on models up to a certain size. Do we know if these settings hold up for much larger systems?
Alex: That's an open question the paper acknowledges. The robustness of these specific settings across different model sizes hasn't been fully explored. It's a reasonable next step for future research.
Sam: And presumably, if the controls could be made automatic — adjusting themselves as the model learns — you wouldn't even need to tune them manually.
Alex: That does seem to be the logical direction. The core insight here is that it's not enough to simply combine good training methods and hope they work together. You have to actively manage how they interact — when each one speaks, how loudly, and when to let the AI take over on its own. That careful management of the feedback loop is what Stable Advantage Fusion is designed to provide. Thanks for listening to ResearchPod.