Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

Building process reward models (PRMs) for LLM agents is notoriously difficult due to the long-horizon, stateful, and often irreversible nature of agentic environments. Traditional methods like human annotation or Monte Carlo estimation are prohibitively expensive or technically infeasible in these settings. This paper investigates whether the standard reinforcement learning (RL) post-training pipeline already contains the necessary information to perform step-level evaluation without requiring additional training or external reward models.

The authors derive a metric called "progress advantage," which is calculated as the log-probability ratio between an RL-trained policy and its reference policy. They prove that this value recovers the optimal advantage function under a general stochastic Markov decision process (MDP). Because this signal is a byproduct of the standard RL post-training process, it is essentially "free" to compute. The authors validate this approach across four model families and five benchmarks, testing its utility in three key inference-time applications: test-time scaling (selecting the best trajectory from multiple candidates), uncertainty quantification (predicting success or failure), and failure attribution (localizing errors in multi-step tasks).

Progress advantage consistently outperforms existing confidence-based baselines and, in many cases, surpasses dedicated, task-specific reward models. In test-time scaling, it significantly boosts task success rates by effectively scoring candidate trajectories. For uncertainty quantification, it provides a superior signal for predicting trajectory outcomes compared to proprietary LLM-as-a-Judge baselines. Furthermore, it successfully localizes error steps in multi-agent systems, approaching the accuracy of methods specifically trained for failure diagnosis. These results demonstrate that the advantage function, which is already central to policy optimization, is a powerful and underutilized tool for inference-time monitoring and evaluation.