Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

Research Approach

This paper introduces a hierarchical control architecture for multi-agent systems that decouples high-level strategic planning from low-level reactive execution. The system uses a pretrained Large Language Model (LLM) as a centralized meta-controller that observes the global game state and selects from a library of four specialized RL skill policies (Navigate, Combat, Secure, and Retreat). These RL policies, trained independently using Proximal Policy Optimization (PPO), handle high-frequency, low-level actions, while the LLM operates at a slower timescale to provide semantic coordination.

Evaluation and Performance

The authors evaluated this hybrid system in a 2v2 King of the Hill environment against two baselines: a hand-crafted Behavior Tree (BT) and a 'Flat' RL agent trained end-to-end without skill decomposition. The LLM+RL system achieved win rates statistically equivalent to the hand-crafted BT and significantly outperformed the Flat RL baseline. Notably, while the Flat RL agent achieved the highest kill-to-death ratio, it failed to win consistently, highlighting that the LLM's ability to coordinate and prioritize objectives like health recovery and zone control was more effective than raw combat performance.

Human Perception

A user study involving 15 participants revealed that 60% of players perceived the LLM+RL agents as the most human-like. Participants cited behavioral adaptability, tactical variability, and the agents' ability to respond to the player's actions as key factors. This suggests that LLM-based orchestration can bridge the gap between rigid, rule-based AI and overly optimized, 'robotic' end-to-end RL, providing a more engaging experience for human players without requiring manual rule engineering.

This research demonstrates a practical path toward building complex, believable multi-agent AI for games. By leveraging the reasoning capabilities of pretrained LLMs to manage pretrained RL skills, developers can avoid the sample inefficiency of training hierarchical policies from scratch and the development burden of writing complex, brittle rule-based systems. It effectively shows that semantic reasoning can be successfully integrated into reactive control loops to improve both strategic coordination and player-perceived believability.