Superhuman AI for Generals.io Using Self-Play Reinforcement Learning

Research Question and Approach

This paper investigates whether a straightforward, well-tuned reinforcement learning (RL) pipeline can achieve superhuman performance in complex real-time strategy (RTS) games without the need for intricate, hand-engineered reward shaping or complex multi-agent league structures. The authors focus on Generals.io, an RTS game characterized by partial observability, long-horizon planning, and tactical combat. To overcome the computational bottlenecks typical of such games, the researchers developed a JAX-native simulator capable of performing tens of millions of frames per second on a single GPU. They trained a vision transformer policy end-to-end using Proximal Policy Optimization (PPO) with sparse win/loss rewards, augmented by a spawn-distance curriculum, parameter exponential moving average (EMA), and top-advantage sample filtering.

The resulting agent reached the #1 position on the public 1v1 leaderboard, outperforming over 5,000 human players by a significant margin. In direct head-to-head matches, the agent defeated the two top-ranked human players with a combined record of 199-70. The study demonstrates that once the data bottleneck is removed by a high-throughput simulator, a standard policy-gradient approach is sufficient to master the game's strategic and tactical complexities. The authors also performed ablations showing that parameter EMA and top-advantage filtering are critical for achieving this level of performance, while other complex regularizers were unnecessary.

This work challenges the prevailing belief that achieving superhuman performance in complex RTS games requires highly specialized, multi-agent training architectures or extensive reward engineering. By showing that a fast simulator and a clean, well-tuned RL loop can suffice, the authors provide a more accessible and reproducible recipe for future research in multi-agent systems and strategic AI. The release of the JAX-native environment further lowers the barrier to entry for researchers interested in cooperative and general-sum multi-agent reinforcement learning.