Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce \textbf{Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization} (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to $5\%$ on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by $3.8\%$ on average and up to $9.2 \%$ on AMC23, and on coding benchmarks it improves pass rate by up to $2.3\%$, while in all settings maintaining the easier objectives near their already satisfied levels.
Alex: Welcome to another episode of ResearchPod. Today we're looking at how artificial intelligence models are trained to handle multiple goals at once — and why the standard approach has a subtle but meaningful flaw.
Sam: So the paper is asking why training systems keep spending time on easy rules they've already mastered, instead of focusing on harder problems.
Alex: Exactly. Think of a student who has perfected their handwriting but still struggles with complex calculus word problems. If you keep giving that student equal practice time on both, most of their effort is wasted on something they've already nailed. That's essentially what's happening inside these AI systems.
Sam: Why can't the system notice that on its own? What's going wrong under the hood?
Alex: The standard approach takes all the different rewards — correctness, formatting, response length — and combines them into a single score. Then it uses that one number to judge each response. But when you collapse everything into one figure, you lose the detail. Two very different responses can end up with the exact same combined score: one where the answer is correct but poorly formatted, and another where the formatting is perfect but the answer is wrong. The system can't tell them apart.
Sam: So the signal gets muddied. You can't see which specific goal is thriving and which is struggling.
Alex: Precisely. And that's the gap this paper addresses. The researchers developed a method called Saturation Aware Advantage Reweighting — SA-MRPO for short. The core idea is straightforward: instead of treating all objectives as equally urgent, the system continuously measures how close each goal is to its maximum possible reward. The closer a goal is to its ceiling, the less training attention it receives.
Sam: How does it actually measure that closeness?
Alex: It looks at recent performance across a batch of responses and estimates what the best achievable score for each objective currently is. If the length reward is consistently hitting near its maximum, the system recognises that goal is saturated — there's very little room left to improve — and it quietly scales back the weight given to that objective. The remaining attention flows toward goals where meaningful progress is still possible.
Sam: It's like a coach who stops drilling the plays the team already executes perfectly, and redirects practice time to the ones they keep getting wrong.
Alex: That's a good way to put it. And the researchers tested this in a concrete setting. They trained a model on math reasoning tasks where responses also had to stay under a specific length. Once the model reliably kept its answers short enough, the length reward maxed out. At that point, SA-MRPO shifted its focus almost entirely toward mathematical accuracy — and accuracy improved meaningfully while response lengths stayed comfortably within the limit.
Sam: Did they test it beyond math?
Alex: They did — coding tasks as well. There, the model had two objectives: get the code to compile, and get it to pass functional test cases. Compilation is relatively straightforward to achieve. Passing every test case is much harder. The method recognised that the compilation goal was saturating quickly and redirected effort toward functional correctness, which is where the real challenge lives.
Sam: Is there a risk of pushing this too far? If the system completely ignores a secondary constraint, couldn't things go wrong?
Alex: That's a real consideration, and the paper addresses it. There's a tuning parameter — think of it as a dial — that controls how aggressively the system discounts saturated objectives. Turn it too high, and the model might start neglecting secondary rules it had already learned. The researchers found that moderate settings strike the best balance: enough reweighting to redirect effort meaningfully, but not so much that those auxiliary constraints get abandoned.
Sam: So it's not a binary switch. You calibrate it carefully.
Alex: Exactly. And that nuance matters, because the broader principle here isn't just about this one method. In any system juggling multiple goals, the question isn't only "how well is each goal being met right now?" It's "how much useful improvement is still available?" Those are different questions, and conflating them leads to wasted effort.
Sam: It's a bit like resource allocation in general. The most valuable place to invest is where there's still room to grow, not where you've already hit a ceiling.
Alex: That's the core insight. The paper also notes an honest limitation: just because a goal appears to have headroom doesn't mean the model can actually exploit it. There are capacity constraints — the model may simply not be capable of improving further on a given objective, regardless of how much attention it receives. The authors flag that as a challenge for future work.
Sam: So the method is a meaningful step forward, but not the final word.
Alex: That's a fair summary. Multi-reward training is a genuinely difficult problem, and this work offers a principled way to think about it — track remaining potential, not just current performance. Thanks for listening to ResearchPod.