OPEN-ENDED MULTI-AGENT AUTOCURRICULA VIA VISUAL INSPECTION OF POLICIES WITH MULTI-MODAL LLMS

In open-ended reinforcement learning, designing a curriculum that effectively sequences tasks is difficult because agents often struggle to identify the next 'stepping stone' skill. Existing methods typically rely on scalar performance metrics (like win rates) or textual summaries, which often fail to capture nuanced behavioral insights. This paper asks whether directly inspecting agent behavior through video—using a Vision Language Model (VLM)—can provide a more effective signal for curriculum generation than traditional, non-visual approaches.

The authors introduce Visual Inspection of Policies (VIP), a framework where a VLM is used to observe recorded episode videos of an agent's current policy. At each curriculum step, the agent is trained on a task, and the resulting behavior is captured as video. This video, along with basic performance metrics, is fed into a VLM, which is prompted to recommend the next most 'interesting' task for the agent to learn. A sentence-similarity module then sanitizes the VLM's output to ensure it maps to a valid task specification within the environment's available task space.

The authors evaluated VIP on the StarCraft Multi-Agent Challenge (SMAC), a complex, multi-agent collaborative environment. They found that VIP consistently outperformed both text-only ablations and traditional curriculum methods that rely on scalar learning signals. The visual data allowed the VLM to identify when agents were close to discovering a winning strategy—even when the win rate was 0%—enabling the curriculum to persist on tasks that were actually productive rather than prematurely switching to others. The authors demonstrate that this approach is computationally efficient, with VLM-based recommendations accounting for only about 1% of total training time.

This work highlights that visual information contains critical, high-level behavioral cues that are often lost when compressing agent performance into scalar values or text. By leveraging the multi-modal capabilities of modern VLMs, researchers can create more intuitive and effective autocurricula that better mimic how a human coach might evaluate an athlete's progress. This approach is particularly promising for multi-agent settings where the interactions between agents are complex and difficult to summarize through simple statistics.