HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

Reinforcement Learning (RL) post-training for Large Language Models is computationally expensive, primarily due to the long-horizon autoregressive sampling required during the rollout phase. While quantization is a standard approach to reduce memory and compute costs, moving to 4-bit precision (FP4) for both the rollout and training policies introduces significant instability. Unlike pretraining, where quantization noise is more manageable, RL post-training involves a continuously shifting policy, making traditional static calibration techniques ineffective.

The researchers conducted a systematic study to isolate the source of performance degradation in fully FP4 RL. They discovered that the primary failure mode is not the training-side quantization, but rather the rollout activation quantization. Specifically, activation outliers stretch the dynamic range of the tensors, causing a large portion of the activation values to underflow to zero. Counterintuitively, the study found that keeping the training policy in higher precision while leaving the rollout in FP4 actually degrades performance further, highlighting a critical rollout-training mismatch that standard pretraining-style fixes cannot resolve.

To address this, the authors propose Rollout Residual Quantization (Rollout-ResQ). This method captures the quantization error (the residual) caused by activation outliers and adds a lightweight, sparse correction term to the rollout matrix multiplication. By constraining this correction to a hardware-friendly sparsity pattern, the method recovers the precision lost to underflow without inflating the compute footprint of the rollout phase. When paired with the HiFloat4 (HiF4) format—which uses a three-level hierarchical scaling scheme to preserve resolution—this approach brings fully quantized FP4 RL performance within 1.1% of the BF16 baseline on Qwen2.5 models.