Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Multi-agent systems (MAS) often struggle to balance high-performance coordination with strict safety requirements. Existing methods typically force a choice between learning-based approaches that lack formal safety guarantees and control-theoretic methods that are computationally expensive or overly conservative. This paper asks whether it is possible to achieve both efficient coordination and hard safety guarantees in a multi-agent setting without the high computational cost of solving optimization problems like quadratic programming (QP) at every timestep.

The authors propose a hierarchical framework called Hierarchical Manifold Multi-Agent PPO (HMM). The architecture decouples the problem into two levels:

1. High-Level Coordination: A centralized training, decentralized execution (CTDE) policy uses graph neural networks to generate subgoals for each agent, facilitating cooperation and path planning.
2. Low-Level Safety: Each agent employs a constraint manifold controller that projects the high-level subgoals onto a safe tangent space. By embedding safety constraints directly into a differentiable manifold, the system ensures that actions remain within safe bounds at every timestep.

This design avoids the need for solving a QP at every step, as the projection onto the tangent space is computationally efficient. Furthermore, because the low-level controller is fixed and non-learnable, the high-level policy experiences stationary transition dynamics, which significantly improves training stability compared to traditional hierarchical reinforcement learning.

The HMM framework provides formal safety guarantees by construction, ensuring that agents remain within a defined safe set throughout both training and deployment. Empirically, the method outperforms existing safety-constrained multi-agent baselines on Lidar-based navigation tasks. Notably, the model demonstrates strong scalability and generalization; policies trained on small scenarios (e.g., 3 agents and 3 obstacles) successfully generalize to much larger environments (up to 21 agents or obstacles) while maintaining near-perfect safety rates.