ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning

Modern large language model (LLM) training often separates inference and training engines to maximize computational efficiency. This separation, combined with the use of low-precision quantization (e.g., FP8) during inference, creates a significant training-inference discrepancy. This mismatch causes the training process to deviate from the intended on-policy learning, leading to instability or total training collapse. The authors seek a way to control this discrepancy to ensure stable training without sacrificing model accuracy.

The authors propose Adaptive Control Reinforcement Learning (ACRL). Instead of forcing the training and inference policies to be identical—which can restrict the parameter space and hurt performance—ACRL treats the training-inference discrepancy as a dynamic variable to be regulated. The framework calculates a reference discrepancy value from the model's initial weights and then uses an Advantage Guided Probability Adjustment (AGPA) mechanism. This mechanism adjusts token-level gradient updates based on whether the current sequence-level discrepancy is above or below the reference, effectively steering the training policy to maintain a stable, optimal distance from the inference policy.

ACRL successfully stabilizes reinforcement learning under aggressive FP8 quantization, a setting where standard methods often fail or require costly precision alignment. By adaptively controlling the discrepancy, the method prevents the training process from collapsing while simultaneously increasing policy entropy, which encourages better exploration. Empirical results on mathematical reasoning benchmarks (such as GSM8K and MATH500) demonstrate that ACRL-trained models achieve accuracy comparable to BF16 baselines and significantly outperform existing importance sampling (IS) fixes.