Yixuan Wang, Yifei Chen, Haichao Zhang, Haozheng Luo, Xander Wu, Jie Ni, Yun Fu, Nuno Vasconcelos, Yijiang Li
5 min
Abstract
Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce \textbf{Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization} (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to $5\%$ on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by $3.8\%$ on average and up to $9.2 \%$ on AMC23, and on coding benchmarks it improves pass rate by up to $2.3\%$, while in all settings maintaining the easier objectives near their already satisfied levels.
Sam: Did they test it beyond math?
Alex: They did — coding tasks as well. There, the model had two objectives: get the code to compile, and get it to pass functional test cases. Compilation is relatively straightforward to achieve. Passing every test case is much harder. The method recognised that the compilation goal was saturating quickly and redirected effort toward functional correctness, which is where the real challenge lives.
Sam: Is there a risk of pushing this too far? If the system completely ignores a secondary constraint, couldn't things go wrong?
Alex: That's a real consideration, and the paper addresses it. There's a tuning parameter — think of it as a dial — that controls how aggressively the system discounts saturated objectives. Turn it too high, and the model might start neglecting secondary rules it had already learned. The researchers found that moderate settings strike the best balance: enough reweighting to redirect effort meaningfully, but not so much that those auxiliary constraints get abandoned.
Sam: So it's not a binary switch. You calibrate it carefully.
Alex: Exactly. And that nuance matters, because the broader principle here isn't just about this one method. In any system juggling multiple goals, the question isn't only "how well is each goal being met right now?" It's "how much useful improvement is still available?" Those are different questions, and conflating them leads to wasted effort.
Sam: It's a bit like resource allocation in general. The most valuable place to invest is where there's still room to grow, not where you've already hit a ceiling.
Alex: That's the core insight. The paper also notes an honest limitation: just because a goal appears to have headroom doesn't mean the model can actually exploit it. There are capacity constraints — the model may simply not be capable of improving further on a given objective, regardless of how much attention it receives. The authors flag that as a challenge for future work.
Sam: So the method is a meaningful step forward, but not the final word.
Alex: That's a fair summary. Multi-reward training is a genuinely difficult problem, and this work offers a principled way to think about it — track remaining potential, not just current performance. Thanks for listening to ResearchPod.