Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

This paper addresses the challenge of reinforcement learning (RL) in continuous-time extended mean field control (MFC) problems. In these settings, the system dynamics and rewards depend on the joint distribution of states and controls, making the problem inherently infinite-dimensional. Traditional approaches often rely on stochastic policies, which require optimizing over complex probability kernels and often lead to unstable or computationally expensive algorithms.

The authors propose a framework using deterministic feedback policies, where the action is a function of the current time, state, and state distribution. By treating the state-action distribution as a push-forward of the state law, they simplify the optimization process. The core of their approach is a model-free sensitivity formula for McKean-Vlasov dynamics, which allows them to derive a policy gradient expressed through an advantage-rate function on the Wasserstein space.

The paper provides a theoretical foundation for learning in extended MFC by:

• Establishing a deterministic policy gradient formula that includes both local action derivatives and measure-derivative terms with respect to the control distribution.
• Developing a martingale-based learning principle that allows agents to learn local value and advantage-rate representations directly from trajectories without knowing the underlying system coefficients.
• Proposing a continuous-time deep deterministic policy-gradient algorithm that integrates measure-dependent neural networks, particle approximations, and temporal-difference learning.

Numerical experiments, including stochastic Cucker-Smale consensus control and optimal liquidation with trade crowding, demonstrate that this deterministic approach is more stable and efficient than existing stochastic-policy methods, particularly when the system dynamics depend explicitly on the control distribution.

This work bridges the gap between theoretical mean field control and practical reinforcement learning. By removing the need for stochastic kernels and providing a robust gradient representation, the authors enable the application of deep RL to complex, large-population systems. This is particularly relevant for financial engineering and multi-agent systems where agents interact through their collective behavior, and where explicit knowledge of the underlying dynamics is unavailable.