RELATIVE VALUE LEARNING

Standard reinforcement learning (RL) typically trains a critic to estimate the absolute value of a state, $V(s)$. However, optimal control decisions depend only on the relative differences between states, not their absolute magnitudes. This paper investigates whether learning these value differences directly—rather than as a derived quantity—can improve the stability and theoretical consistency of RL agents.

The authors propose Relative Value Learning (RV), a framework that learns an antisymmetric function $\Delta(s_i, s_j) = V(s_i) - V(s_j)$. By defining a pairwise Bellman operator, the authors prove that this function is a $\gamma$-contraction with a unique fixed point corresponding to the true value differences. They also derive a Relative Generalized Advantage Estimation (R-GAE) method, which allows for unbiased policy-gradient updates without requiring absolute state values. To handle the practical challenge of trajectory-specific offsets, the authors introduce a trajectory-ranking mechanism that aligns relative values across different episodes.

The theoretical framework successfully eliminates the 'gauge freedom' inherent in absolute value estimation, where arbitrary shifts in value do not affect policy performance. Empirically, the authors integrated RV into the Proximal Policy Optimization (PPO) algorithm. Testing across 49 Atari games in the Arcade Learning Environment (ALE), the PPO+RV agent demonstrated competitive performance against standard PPO, validating that relative value estimation is a viable and robust alternative to traditional absolute critics.

This work shifts the focus of value-based RL from absolute magnitude estimation to relational learning. By aligning the learning objective with the actual invariants of decision-making, the framework provides a more principled way to handle reward shaping and baseline changes. Furthermore, it offers a promising path for settings where absolute values are ambiguous or unavailable, such as in preference-based or human-in-the-loop RL.