MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

As multimodal large language models (MLLMs) are increasingly deployed as embodied agents, a critical question arises: how do these models perform when required to collaborate in visually grounded, dynamic environments? While single-agent performance has been extensively studied, multi-agent collaboration introduces complex challenges—such as spatial conflicts, communication overhead, and the need for distributed exploration—that are not captured by text-only benchmarks. This paper introduces MECoBench to systematically evaluate these collaborative capabilities.

MECoBench provides an evaluation platform built on the VirtualHome simulator, featuring 192 test cases across eight real-world household task templates. The benchmark supports two distinct cooperation structures: parallel (where agents work independently in a shared space) and sequential (where agents are constrained to specific zones and must perform object handovers). The researchers evaluated a diverse set of 14 MLLMs, ranging from 8B to 235B parameters, across varying team sizes (1–5 agents), collaboration modes (isolated, centralized, and decentralized), and communication mechanisms (textual, shared memory, and vision-augmented).

• Inverted-U Performance Trend: Collaboration consistently improves task completion, but performance follows an inverted-U curve as team size increases. Moderate team sizes optimize the trade-off between parallel task execution and coordination overhead; beyond this, performance saturates and eventually degrades.
• Communication as a Catalyst: Explicit communication is essential for success, particularly in sequential tasks requiring spatial coordination. Centralized leadership is most effective for small teams, while decentralized protocols scale better for larger groups.
• Robustness Under Uncertainty: Multi-agent teams demonstrate superior robustness when prior information is missing or noisy. Distributed exploration and communication allow teams to compensate for misleading priors, yielding the largest relative performance gains compared to single-agent baselines.
• Failure Modes: The study identifies two critical failure patterns: duplicate actions (e.g., multiple agents grabbing the same object) and hallucinated beliefs (e.g., an agent falsely reporting a task as complete, causing the entire team to stop prematurely).