Haipeng Luo, Qingfeng Sun, Songli Wu, Can Xu, Wenfeng Deng, Han Hu, Yansong Tang
5 min
Reinforcement Learning with Verifiable Rewards (RLVR), such as Group Relative Policy Optimization (GRPO), is a standard approach for training complex reasoning in Large Language Models. However, these models frequently suffer from policy entropy collapse, where output diversity vanishes and the policy converges prematurely, limiting the effectiveness of long-horizon training. This paper investigates the underlying mechanism of this collapse and proposes a principled intervention to maintain stable training.
The authors perform a first-order gradient analysis of token-level entropy dynamics. They discover a credit assignment mismatch: while GRPO assigns a single trajectory-level advantage to all tokens, the actual entropy contribution of a token depends on its surprisal and the next-token distribution. This leads to a four-quadrant structure where low-surprisal tokens (the majority) drive entropy reduction, while high-surprisal tokens (the minority) that could maintain diversity are under-represented. To fix this, the authors introduce STARE (Surprisal-guided Token-level Advantage Reweighting), which identifies entropy-critical tokens using batch-internal surprisal quantiles and selectively reweights their advantages. A closed-loop gate ensures this reweighting only activates when the batch-averaged entropy falls below a target threshold.
STARE successfully sustains stable RL training over thousands of steps across various model scales (1.5B to 32B) and task types, including Short CoT, Long CoT, and multi-turn tool use. By preventing entropy collapse, the method maintains a better exploration-exploitation balance, resulting in a 4%-8% improvement in average accuracy on AIME24 and AIME25 benchmarks compared to baselines like DAPO. The authors demonstrate that their approach is robust and requires only minimal modifications to the standard GRPO objective.
As RL post-training becomes the dominant paradigm for reasoning models, the ability to train for longer durations without performance degradation is critical. STARE provides a theoretically grounded, low-intrusion solution to the common problem of premature policy convergence, allowing researchers to scale RL training more effectively.
Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.
Alex: So it's a kind of rebalancing — actively amplifying the signal from rare, creative choices?
Sam: That's a fair way to put it. And crucially, it only does this when the model is genuinely succeeding — when it's getting rewards. The idea is to reinforce the creative paths that led to good outcomes, not just any random output.
Alex: But if you're constantly boosting rare words, doesn't that risk pushing the model toward nonsense? Rare doesn't always mean useful.
Sam: That's exactly the right concern, and the researchers address it directly. They use what they call a "target-entropy gate." Think of it as a thermostat. The system monitors how diverse the model's outputs are. If diversity is healthy, the gate stays closed — STARE does nothing. It only switches on when the model's variety drops below a certain threshold, when it's genuinely starting to collapse into repetition. So it's not constantly interfering. It steps in only when the model is starting to get stuck.
Alex: A closed-loop system. It keeps the model in a kind of Goldilocks zone — varied enough to keep learning, focused enough to stay useful.
Sam: Exactly. Not so repetitive that it stops improving, but not so random that it loses the point of the task entirely.
Alex: And does this actually translate into better performance on real problems?
Sam: The results are notable. On tests like the AIME — a genuinely difficult maths competition — models trained with STARE outperformed standard versions. More meaningfully, they kept improving for thousands of training steps longer than models without it. Standard training tends to plateau relatively quickly. STARE extends that learning window significantly.
Alex: So the gain isn't just a one-time bump — it's that the model stays in a productive learning state for longer.
Sam: That's the core claim, yes. And it points toward something the researchers describe as more sustained training — the idea that with the right entropy management, models could continue improving their reasoning without hitting the kind of premature ceiling that current methods tend to produce. The paper is careful not to overstate this, but the direction is clear.
Alex: It's a thoughtful piece of work. The insight that the training process itself can undermine what you're trying to build — and that a relatively targeted fix can extend learning so substantially — is worth sitting with. Thanks for walking us through it, Sam.
Sam: Thanks, Alex.
Alex: That's it for today's episode. Thanks for listening to ResearchPod.