The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning | Jing Liang et al. | ResearchPod

Transcript: The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a specific challenge in how we train large language models using reinforcement learning.

Sam: We're discussing a paper that tackles what researchers call "training-inference mismatch." Here's the core issue: the software used to train an AI model and the software used to actually run it in the real world are often two different systems. Even when they're designed to be identical, they can interpret the model's instructions in subtly different ways — and that gap causes unstable, unpredictable performance.

Alex: So the model might learn to perform well in one environment, but then stumble when it's actually deployed?

Sam: Exactly. Think of it like a student who practices for an exam in one room, then sits the actual test in a room with different lighting, different desks, different acoustics. The knowledge is the same, but the environment throws them off. Current training methods assume that if the model improves during practice, it will automatically improve in the real world. This paper shows that assumption isn't guaranteed.

Alex: And the paper proposes a fix?

Sam: It does. The framework is called Monotonic Inference Policy Update — MIPU. The key idea is to treat the deployment system, what's called the inference engine, as the final judge of whether any training update is actually worth keeping.

Alex: How does that work in practice?

Sam: It's a two-step process. Step one: generate a candidate update to the model, but base it on how the inference engine actually generates answers — not just the training engine's version. That way, the practice questions more closely match the real test. Step two: before accepting that update, run a check. Calculate how differently the training engine and the inference engine view the proposed change. If the update looks good in the training room but would cause a negative shift in the deployment room, the system rejects it outright.

Alex: So it's a quality control filter. It only keeps updates that genuinely improve the model where it matters.

Sam: Right. And there's an important subtlety here. It's not just throwing away anything that isn't perfect. The framework guarantees that every accepted update is at least as good as the previous one, from the deployment engine's perspective. Think of it like a safety harness on a climbing wall. The model is free to explore, but if a move would send it off a cliff, the harness catches it and returns it to the last stable position.

Alex: So the model never takes a step backward. How does this compare to the standard approaches?

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng

Jun 28, 2026·5:04·arxiv·Original Paper

Training-Inference MismatchMonotonic Inference Policy Improvement (MIPI)Monotonic Inference Policy Update (MIPU)Objective MisalignmentMIPU (Monotonic Inference Policy Update)MonotonicInferencePolicyImprovement (MIPI)

Listen

5 min

Briefing

Research Question

Modern LLM reinforcement learning (RL) pipelines often use separate engines for training (high precision) and inference (high efficiency). This architectural split leads to a training-inference mismatch, where the training policy and the inference policy assign different probabilities to the same trajectories, even when model parameters are synchronized. The authors investigate whether optimizing the training policy—the standard approach—is actually the correct objective when the inference policy is the one used for deployment.

Approach

The authors propose a new principle called Monotonic Inference Policy Improvement (MIPI). Instead of focusing on the training-side surrogate, MIPI targets the monotonic improvement of the inference policy directly. To implement this, they introduce the Monotonic Inference Policy Update (MIPU) framework, which operates in two steps:

Sampler-Referenced Policy Update: The training engine performs an update that accounts for the sampler (inference) policy, ensuring that the candidate update is theoretically beneficial relative to the current inference policy.
Inference-Gap-Aware Acceptance: The synchronized candidate is evaluated using a proxy that measures the post-update gap between the training and inference engines. If the update does not meet a specific performance criterion, the system rejects the update and rolls back to the previous checkpoint.

Findings

Experiments conducted under high-mismatch conditions (using FP8-quantized rollouts) demonstrate that MIPU significantly improves both reasoning performance and training stability compared to standard RL baselines. The authors show that the two-step process effectively decouples the construction of candidate updates from the validation of their deployment-side performance, preventing the model from accepting updates that appear beneficial during training but degrade performance during inference.

Key Terms and Definitions

Training-Inference Mismatch — The phenomenon where the training engine and inference engine assign different probabilities to the same trajectories due to differences in precision, decoding, or serving backends.
Monotonic Inference Policy Improvement (MIPI) — A principle stating that the RL objective should be defined to ensure the inference policy improves monotonically, rather than just the training policy.
Sampler-Referenced Update — A policy update method that uses the inference policy (the sampler) as the reference point for calculating probability ratios and advantages, rather than the training policy.
Inference-Gap-Aware Acceptance — A validation mechanism that checks if a candidate model update actually improves the inference engine's performance before committing the change.
Objective Misalignment — The condition where an update that increases the reward on the training side does not necessarily result in an increase in reward on the inference side.

Abstract

Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning: Research Paper Podcast