Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning

In multi-objective reinforcement learning (MORL), agents must balance conflicting objectives. Standard approaches often aggregate these into a single scalar reward, which can fail to capture the full range of optimal trade-offs (the Pareto frontier). This paper addresses the challenge of synthesizing a complete and parsimonious set of deterministic policies that accurately represent this frontier without the computational burden of storing long action histories.

The authors propose a novel preference-conditioned Bellman operator motivated by Chebyshev scalarization. By parameterizing the value function with a preference vector, the agent can target specific trade-offs between objectives. The operator effectively identifies the 'bottleneck' objective for a given preference, allowing the agent to optimize along that direction. A key innovation is the proof that this operator satisfies an enveloping property—where estimates upper-bound the true Pareto frontier—and that it converges to a coverage set of the frontier. To extract policies, the authors demonstrate that an agent can dynamically update its preference parameter based on the current state and action, eliminating the need for complex, history-dependent policy representations.

The paper establishes that this preference-aware Bellman update allows for the recovery of deterministic, Pareto-optimal policies. The authors prove that their method converges asymptotically to the Pareto-optimal values and that the resulting policy set provides an approximate coverage of the entire Pareto frontier. Empirical results confirm that the algorithm successfully recovers complex trade-offs in multi-objective environments, providing a scalable solution for deterministic Pareto-optimal policy synthesis.

This work provides a formal framework for multi-objective decision-making that bridges the gap between theoretical optimality and practical implementation. By enabling the synthesis of deterministic policies that cover the Pareto frontier, the approach offers a robust way to handle real-world scenarios—such as circuit design, robotics, or resource allocation—where the relative importance of objectives may shift depending on the user's needs or the operating context.