Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies

The softmax policy, defined as pi(a|s) proportional to exp(beta * Q(s,a)), is the standard model for stochastic choice in reinforcement learning (RL). Despite its ubiquity in algorithms like Soft Actor-Critic and RLHF, it has historically lacked a derivation from normative first principles. This paper addresses the tension between the entropy-regularized Bellman equation and the standard Independence axiom of expected utility theory.

The author resolves this conflict by distinguishing between two types of randomness: chance and choice. The paper restricts the traditional von Neumann-Morgenstern (VNM) Independence axiom to environmental lotteries (chance), while applying Independence of Irrelevant Alternatives (IIA) and monotonicity to behavioral selection (choice). By treating the entropy bonus as an 'option premium'—the value of retaining the ability to choose—the author proves that these axioms uniquely determine the Boltzmann policy and the soft Bellman equation.

The central result (Theorem 3.1) establishes that for a finite MDP, an agent satisfying VNM preferences over base prospects, along with IIA and monotonicity, must follow a Boltzmann policy. The entropy bonus emerges naturally from these axioms rather than being an ad-hoc addition. The paper also demonstrates that the soft Bellman equation is the only structure consistent with these axioms, and it identifies a minimum rationality threshold (beta_0) below which the soft Bellman equation may fail to converge without additional continuity assumptions.

This work provides a normative foundation for entropy-regularized RL, moving it from an empirical design heuristic to a principled choice. It clarifies that using softmax is equivalent to assuming that an agent values its own ability to choose according to the IIA property. This framework helps researchers identify when softmax is appropriate (e.g., independent actions) and when it might be theoretically misaligned (e.g., when actions are correlated or redundant).