VIMPO: Value-Implicit Policy Optimization for LLMs

Research Question and Approach

Reinforcement Learning with Verifiable Rewards (RLVR) is essential for improving LLM reasoning, but practitioners face a trade-off: actor-critic methods provide dense, token-level supervision but suffer from the instability of training a separate value function (critic), while group-relative methods like GRPO are stable and critic-free but rely on coarse, trajectory-level advantages. VIMPO (Value-Implicit Policy Optimization) aims to bridge this gap by deriving a value function directly from the optimality conditions of KL-regularized reinforcement learning. By modeling autoregressive generation as a deterministic-transition Markov Decision Process (MDP), the authors show that the optimal value function can be expressed through policy-reference log-ratios. This allows the model to compute a Bellman-consistent value loss and a one-step advantage without ever training a separate neural network as a critic.

VIMPO outperforms GRPO across several challenging mathematical reasoning benchmarks, including MATH-500, AIME 2024, AIME 2025, and OlympiadBench. The authors demonstrate that VIMPO achieves faster training convergence and higher final validation accuracy compared to GRPO baselines. Notably, the performance gains are most significant on harder, competition-style problems, suggesting that VIMPO's finer credit assignment is particularly beneficial for complex reasoning tasks. Furthermore, in stress tests using noisy reward signals, VIMPO maintains a consistent performance advantage over GRPO, indicating that the policy-implied value optimization is more robust to imperfect reward feedback.

VIMPO provides a theoretically grounded way to achieve the benefits of actor-critic methods—specifically dense, token-level credit assignment—while retaining the practical simplicity and stability of critic-free training. By eliminating the need for a learned critic, it removes a significant source of training instability and hyperparameter sensitivity in LLM post-training. This approach offers a scalable path for improving reasoning capabilities in models where training a reliable value function is otherwise difficult or computationally expensive.