Yixuan Wang, Yifei Chen, Haichao Zhang, Haozheng Luo, Xander Wu, Jie Ni, Yun Fu, Nuno Vasconcelos, Yijiang Li
5 min
Reinforcement learning with verifiable rewards (RLVR) has become standard for post-training language model reasoners, often relying on group-relative advantage estimation without a learned value function. However, practical reasoning models must satisfy multiple objectives simultaneously, such as correctness, length constraints, format compliance, and safety. Existing approaches typically combine these objectives into a fixed weighted sum before group-wise standardization. This scalarization introduces two fundamental problems: distinct reward profiles can collapse to identical scalar advantages, and all objectives are optimized with fixed relative weights regardless of their current saturation level. Consequently, training wastes gradient budget on already-solved objectives instead of focusing on those with greater remaining headroom.
To overcome these limitations, the authors introduce Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO). SA-MRPO preserves per-objective normalization while dynamically discounting each objective's contribution based on how close it is to its maximum attainable reward. A single hyperparameter controls this saturation-aware reweighting, allowing the optimization process to automatically shift focus toward under-optimized objectives.
SA-MRPO modifies the advantage construction step of group-relative policy optimization while keeping the underlying policy update and surrogate objective intact. For each objective, the method computes group statistics and measures the saturation ratio using the batch-mean reward relative to the objective's lower and upper bounds. Objectives that are closer to their reward ceilings receive progressively smaller weights, effectively reducing their gradient influence.
Mathematically, this reweighting does more than just scale down magnitudes; it can actively reverse the sign of an update depending on the aggregate rollout advantage. SA-MRPO also strictly generalizes existing baselines: disabling saturation-aware reweighting reduces it to Group reward-Decoupled Policy Optimization (GDPO), and further reducing it to a single objective yields standard Group Relative Policy Optimization (GRPO).
Across multiple domains, SA-MRPO demonstrates consistent performance gains over existing multi-objective baselines:
Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce \textbf{Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization} (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to $5\%$ on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by $3.8\%$ on average and up to $9.2 \%$ on AMC23, and on coding benchmarks it improves pass rate by up to $2.3\%$, while in all settings maintaining the easier objectives near their already satisfied levels.
Sam: Did they test it beyond math?
Alex: They did — coding tasks as well. There, the model had two objectives: get the code to compile, and get it to pass functional test cases. Compilation is relatively straightforward to achieve. Passing every test case is much harder. The method recognised that the compilation goal was saturating quickly and redirected effort toward functional correctness, which is where the real challenge lives.
Sam: Is there a risk of pushing this too far? If the system completely ignores a secondary constraint, couldn't things go wrong?
Alex: That's a real consideration, and the paper addresses it. There's a tuning parameter — think of it as a dial — that controls how aggressively the system discounts saturated objectives. Turn it too high, and the model might start neglecting secondary rules it had already learned. The researchers found that moderate settings strike the best balance: enough reweighting to redirect effort meaningfully, but not so much that those auxiliary constraints get abandoned.
Sam: So it's not a binary switch. You calibrate it carefully.
Alex: Exactly. And that nuance matters, because the broader principle here isn't just about this one method. In any system juggling multiple goals, the question isn't only "how well is each goal being met right now?" It's "how much useful improvement is still available?" Those are different questions, and conflating them leads to wasted effort.
Sam: It's a bit like resource allocation in general. The most valuable place to invest is where there's still room to grow, not where you've already hit a ceiling.
Alex: That's the core insight. The paper also notes an honest limitation: just because a goal appears to have headroom doesn't mean the model can actually exploit it. There are capacity constraints — the model may simply not be capable of improving further on a given objective, regardless of how much attention it receives. The authors flag that as a challenge for future work.
Sam: So the method is a meaningful step forward, but not the final word.
Alex: That's a fair summary. Multi-reward training is a genuinely difficult problem, and this work offers a principled way to think about it — track remaining potential, not just current performance. Thanks for listening to ResearchPod.