EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games

In two-player zero-sum imperfect-information games, self-play training often relies on regularization to stabilize learning and prevent the policy from collapsing into suboptimal strategies. Standard approaches typically use a uniform distribution as a fixed regularization target. However, this approach is strategically agnostic—it forces the agent to spend its regularization budget on all actions equally, even those that are strictly dominated. This paper investigates whether an adaptive regularization target, which evolves alongside the agent's strategy, can improve performance in complex games with large strategy spaces.

The authors introduce EMAgnet, a method that replaces the static uniform regularization target with an exponential moving average (EMA) of the policy's own neural network parameters. By maintaining this parameter-space EMA, the regularization target effectively "remembers" viable strategies encountered during training while gradually "forgetting" strategies that the agent has learned to avoid. This approach is integrated into the Proximal Policy Optimization (PPO) algorithm, requiring only a simple weight update after each training step.

EMAgnet was evaluated against PPO baselines using both linear and power-law annealing schedules across standard benchmarks (e.g., Kuhn Poker, Goofspiel) and modified versions containing strictly dominated strategies (Forfeit and Control games). The results demonstrate that:

• EMAgnet matches or outperforms uniform-magnet baselines on standard games.
• In games with strictly dominated strategies, EMAgnet consistently achieves lower exploitability and faster convergence.
• Unlike uniform regularization, which suffers from a dilemma between wasting budget on dominated actions or losing necessary exploration pressure, EMAgnet maintains constant regularization strength while dynamically shifting its focus toward strategically relevant actions.