What are Key Factors for Updates in RL for LLM Reasoning?

Research Context and Motivation

Reinforcement Learning from Verifiable Rewards (RLVR) has become a standard approach for improving the reasoning capabilities of Large Language Models (LLMs). However, the field has been characterized by heuristic-driven algorithmic choices—such as prioritizing high-entropy tokens versus high-probability tokens—that often yield contradictory empirical results. This paper seeks to move beyond these heuristics by providing a unified theoretical analysis of how RLVR updates behave under different training conditions.

Theoretical Analysis of Gradient Dynamics

The authors identify the "off-policy degree" (the number of gradient updates performed per rollout) as the primary factor driving these contradictions. By analyzing the expected gradient under clipped importance sampling, they demonstrate that the variance of the importance sampling (IS) ratio is highly sensitive to token probability. Specifically, low-probability tokens exhibit much higher IS-ratio variance, making them more susceptible to clipping as the training moves further off-policy. This leads to a "gradient dominance reversal": in near on-policy regimes, low-probability tokens dominate the update, while in highly off-policy regimes, they are frequently clipped, causing high-probability tokens to dominate instead. This finding reconciles previous, seemingly conflicting studies.

Adaptive Clip Policy Optimization (ACPO)

Building on this insight, the authors propose Adaptive Clip Policy Optimization (ACPO). Standard RLVR methods typically apply a single, global clipping window to all tokens. ACPO replaces this with bin-wise clipping thresholds that adapt to the empirical variance of the IS ratio for different token groups. By adjusting these boundaries, ACPO prevents the uneven suppression of gradient signals and mitigates the directional bias introduced by standard clipping. Experimental results across 3B and 7B models on mathematical, tabular, and logic-based reasoning tasks show that ACPO consistently outperforms strong baselines like DAPO and CISPO.

This work shifts the paradigm of RLVR research from trial-and-error heuristics to a principled, analysis-driven framework. By mathematically characterizing how clipping interacts with token probability and off-policy degree, the authors provide a robust mechanism for stabilizing LLM training. This is particularly important for practitioners who must balance the computational efficiency of reusing rollouts (off-policy) with the need for stable, high-quality gradient updates.