Generalised Bellman recurrence and three dualities in sequential decision-making

The Bellman equation is the cornerstone of sequential decision-making, yet its origins are often treated as a primitive principle. This paper formalizes the Bellman equation as a consequence of three structural conditions:

1. Condition D (History Sufficiency): Dynamics must decompose through a sufficient statistic, allowing the transition to depend only on the current state rather than the entire history.
2. Condition R (Return Compositionality): The return functional must be decomposable into a local reward and a tail return, linked by an associative operator.
3. Condition A (Aggregation Compatibility): The method used to aggregate uncertainty must be compatible with the recursive structure of both the dynamics and the returns.

When these three conditions hold on a common state, the Bellman equation emerges naturally. If one condition is violated, the authors demonstrate that tractability can often be restored by augmenting the state space or by deforming the return or dynamics to re-establish the necessary structure.

The authors identify three dualities that arise from these conditions, which explain the relationships between seemingly disparate methods in reinforcement learning and control:

• Object Duality (Statistics vs. Returns): Relates probabilities and returns, showing that maximizing expected return is symmetric with respect to these two objects.
• Assessment Duality (Return vs. Aggregation): Shows that non-linear aggregation (e.g., risk-sensitive or robust control) can be mathematically relocated into the return function through regularisation or curvature adjustments.
• Representation Duality (Dynamics vs. Aggregation): Explains how non-linear aggregators (like CVaR) can be absorbed into a reweighting of the probability distribution, effectively turning a non-standard optimization problem into an expectation-based one.

This framework provides a unified language for sequential decision-making. By mapping various approaches—such as robust control, soft actor-critic, and active inference—onto these three conditions and dualities, the paper reveals that these methods are not isolated techniques but rather specific ways of managing the trade-offs between dynamics, returns, and uncertainty. This allows researchers to systematically derive new algorithms by manipulating these components.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.