Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful technique for improving LLM reasoning, but it often suffers from optimization instability. Standard approaches apply training signals uniformly across all tokens, which can lead to 'entropy collapse' (where the model prematurely converges to suboptimal, deterministic strategies) or 'entropy explosion' (where the model generates incoherent reasoning chains due to excessive, blind exploration). This paper investigates whether focusing updates on specific, high-information tokens can resolve this dichotomy and stabilize the training process.

The authors introduce the Independent Combinatorial Tokens (ICT) framework, which shifts the focus from scalar uncertainty metrics (like Shannon Entropy) to the distributional properties of token logits. By calculating the Jensen-Shannon (JS) divergence between individual token logit distributions and the group-average distribution, the framework identifies 'unique' tokens that serve as critical branching points in reasoning trajectories. The authors then implement a sparse training objective that updates only the top 10% of these unique tokens, effectively masking the gradients for the remaining 90% of the sequence.

Theoretical analysis demonstrates that updating these distributionally unique tokens regulates policy concentration, simultaneously reducing overall distribution uncertainty and controlling probability mass. Empirically, the ICT framework significantly outperforms standard baselines like GRPO, 20-Entropy, and STAPO. Across seven benchmarks covering math, commonsense, and Olympiad-level problems, the sparse update strategy on Qwen2.5 models (0.5B to 7B parameters) yielded an average pass@4 improvement of 4.58%, with peak gains reaching 14.9%. These results suggest that a small subset of tokens drives the majority of performance gains in RLVR.