MEAN-FIELD PHIBE: CONTINUOUS-TIME MEAN-FIELD REINFORCEMENT LEARNING FROM DISCRETE-TIME DATA

In continuous-time reinforcement learning, agents often operate in environments governed by McKean-Vlasov stochastic differential equations (SDEs), but data is typically collected at discrete intervals. Standard methods either ignore the underlying differential structure by reducing the problem to a discrete-time Markov Decision Process (MDP) or fail to account for the non-identifiability of drift and diffusion coefficients when only discrete-time transitions are available. This paper asks: how can we perform model-free mean-field control that preserves the continuous-time structure of the population dynamics while using only discrete-time data?

The authors propose Mean-Field-PhiBE (MF-PhiBE), a framework that bridges the gap between discrete-time data and continuous-time control. Instead of forcing a discrete-time Bellman formulation, the authors incorporate discrete-time transition statistics into a continuous-time PDE defined on the Wasserstein space. By replacing the unknown infinitesimal drift and covariance coefficients in the HJB equation with one-step estimators derived from data, the method maintains the generator structure of the McKean-Vlasov dynamics. This is paired with an entropy-regularized policy gradient theorem to create a model-free actor-critic algorithm.

The study establishes that the MF-PhiBE approach provides a consistent approximation of the optimal continuous-time value function, with an error of order Δt. In the specific case of linear-quadratic mean-field control, the approximation achieves second-order accuracy. Numerical experiments on LQR benchmarks and crowd-aversion problems demonstrate that the MF-PhiBE policy remains significantly closer to the theoretical optimal continuous-time value than policies derived from standard time-discrete Bellman formulations.

This work provides a principled way to handle continuous-time mean-field control problems when the underlying physical model is unknown. By preserving the continuous-time PDE structure, the method avoids the discretization biases inherent in standard MDP-based reinforcement learning, offering a more robust approach for large-population systems where the evolution of the probability law is critical.