ROBUST Q-LEARNING FOR MEAN-FIELD CONTROL UNDER WASSERSTEIN UNCERTAINTY IN COMMON NOISE

Mean-field control (MFC) problems involve optimizing the behavior of a large population of agents, where individual actions are influenced by the aggregate distribution of the population. When these systems are subject to common noise—randomness affecting all agents simultaneously—the problem becomes computationally intractable due to the infinite-dimensional nature of the probability measure space. This paper addresses the challenge of solving these MFC problems when the law of the common noise is unknown and subject to Wasserstein uncertainty, meaning the true distribution lies within a Wasserstein ball around a reference measure.

The authors propose a robust Q-learning algorithm designed to learn the optimal Q-function in this uncertain environment. Because the lifted state space (the space of probability measures) and the set of policies are infinite-dimensional, the authors employ a quantization-and-projection scheme to create a finite-dimensional, tabular representation. To handle the robust optimization component—the worst-case expectation over the Wasserstein uncertainty set—they apply Wasserstein duality theory. This transformation converts the difficult min-max problem into a more manageable optimization problem involving a single expectation under the reference measure, which is suitable for sample-based learning.

The paper establishes that the proposed robust Q-learning algorithm converges to the optimal Q-function, with the error bounded by the discretization granularity. The authors provide both synchronous and asynchronous versions of the algorithm and derive non-asymptotic finite-time iteration bounds, specifying the number of iterations required to reach a target accuracy at a given confidence level. Numerical experiments on systemic risk and epidemic control models demonstrate that the algorithm effectively manages the robustness-performance tradeoff when the common-noise law is misspecified.