Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning

In cooperative multi-agent reinforcement learning (MARL), sequential update frameworks (like HAPPO) allow agents to update their policies one by one, which theoretically guarantees monotonic improvement. However, these methods rely on estimating a joint advantage function using importance sampling. The authors investigate a critical failure mode: the variance of this advantage estimator grows exponentially with the number of agents, leading to unstable training and poor convergence.

The authors provide a theoretical and empirical analysis demonstrating that the importance sampling ratio used in sequential updates is the primary source of variance. To mitigate this, they propose a novel surrogate objective function that applies a clipping mechanism to the importance ratio. This modification constrains the advantage fluctuation during updates. They derive two practical algorithms, clip-HAPPO and clip-HATRPO, and prove that the new objective maintains a monotonic improvement bound and converges to an epsilon-Nash equilibrium at a sub-linear rate.

• The study proves that the variance of the standard advantage estimator in sequential MARL scales exponentially with the number of agents, a phenomenon confirmed by experiments on Ant-v2 and Humanoid-v2 benchmarks.
• The proposed clipping objective effectively suppresses this variance, keeping it stable regardless of the number of agents.
• Experimental results show that the clipped objective leads to more stable training and outperforms standard HAPPO and other baselines across multiple popular MARL benchmarks.