Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards

Large language models often generate excessively long reasoning traces, incurring high computational costs. While researchers have attempted to mitigate this by adding length-penalizing rewards to the Group Relative Policy Optimization (GRPO) algorithm, these methods frequently suffer from reward collapse—a failure mode where the model suddenly produces extremely short, low-quality outputs. This paper investigates the mechanistic root of this instability to enable stable efficiency training.

Through a systematic evaluation of various reward configurations, the authors identify two distinct pathways for reward collapse:

1. The Beta-Loop (Structural Collapse): When length penalties are applied to incorrect answers, GRPO’s group normalization creates a structural bias. In mixed-outcome batches, this results in a constant negative gradient that erodes reasoning length. In homogeneous batches (where all samples are incorrect), the normalization denominator can approach zero, leading to explosive, runaway gradient updates that force the model to truncate reasoning.
2. Stochastic Correct-Answer Compression: Even when penalties are restricted to correct answers, models remain vulnerable to a secondary failure. If the model is incentivized to be concise, it may over-compress correct reasoning traces, leading to a sudden, stochastic drop in accuracy.

To address these issues, the authors introduce Adaptive Correct-Only Efficiency Reward (ACOER). ACOER employs three synergistic mechanisms:

• Correct-Only Signal: By setting the incorrect-answer penalty to zero, it eliminates the structural beta-loop entirely.
• Adaptive Budget Normalization: Instead of using a fixed maximum token limit, ACOER uses a dynamic budget that tracks the exponential moving average of the model's own reasoning lengths, preventing the efficiency signal from becoming overly aggressive as the model improves.
• Control-Loop Penalty Adaptation: The model dynamically adjusts the strength of the efficiency reward based on real-time accuracy, backing off if reasoning performance begins to degrade.

Empirical results on benchmarks like MATH-500 show that ACOER reduces token generation by over 60% while maintaining accuracy comparable to the base model, providing a robust framework for efficient reasoning.