Offline Multi-agent Continual Cooperation via Skill Partition and Reuse

In cooperative multi-agent reinforcement learning (MARL), agents often face the challenge of learning from sequential tasks in offline settings. Existing methods typically rely on fixed-size skill libraries, which struggle with distributional shifts and interference as new tasks emerge, leading to catastrophic forgetting and loss of plasticity. This paper asks: how can agents continuously discover, store, and reuse coordination skills from offline datasets in an open-ended multi-agent environment?

The authors propose COMAD (Continual Offline Multi-agent Skill Discovery via Skill Partition and Reuse). The framework operates in two primary stages for each incoming task. First, it uses a variational auto-encoder (VAE) to discover task-specific coordination skills from offline behavior data, transforming coordination knowledge into reusable latent representations. Second, it employs a multi-head architecture to store these skills. A density-based reusability estimator evaluates whether previously acquired skills are relevant to the current task; if a skill is deemed reusable, it is used to guide the policy learning objective via a skill-augmented advantage function. This allows the agent to expand its skill library dynamically while maintaining performance on previously learned tasks.

Theoretical analysis demonstrates that COMAD approximates the optimal solution for continual skill discovery. Empirical results across diverse MARL benchmarks show that COMAD effectively mitigates interference between tasks. By dynamically expanding the skill library and selectively reusing past coordination patterns, the framework achieves superior forward and backward transfer compared to existing baselines, demonstrating robustness in handling sequential task streams.