MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?

Research Overview

Multi-agent systems (MAS) are increasingly used to solve complex, organization-scale tasks by coordinating multiple LLM-based agents. While system prompts are a primary lever for improving these systems without fine-tuning, optimizing them in a multi-agent context is challenging due to the exponential growth of the search space and complex inter-agent dependencies. This paper introduces MAS-PromptBench, a comprehensive benchmark designed to systematically evaluate prompt optimization across diverse MAS configurations, including varying tasks, workflow topologies, communication protocols, and team sizes.

The authors evaluate the effectiveness of prompt optimization by extending two state-of-the-art single-agent optimizers (GEPA and MIPRO) to the multi-agent setting. Using MAS-PromptBench, they measure the performance delta between default system prompts and optimized prompts across nine tasks spanning reasoning, coding, and tool-calling. The study isolates the impact of four key factors: task domain, workflow topology (e.g., sequential vs. centralized), communication protocol (e.g., freeform vs. structured), and team size.

Key Findings

The results demonstrate that prompt optimization is a powerful tool for MAS, with potential gains reaching 24 percentage points in specific configurations like sequential workflows. However, the study reveals that these gains are not universal. Performance is highly sensitive to the underlying MAS architecture; in some configurations, the same optimization methods caused performance to drop by as much as 16 percentage points. The authors find that optimization is most effective when tasks involve explicit, verifiable agent behaviors and when communication protocols impose a clear, shared structure. As team size increases, the difficulty of optimizing prompts grows, highlighting the need for more robust, scalable, and topology-aware optimization algorithms.

As agentic AI moves toward more complex, multi-agent architectures, relying on manually crafted prompts becomes a bottleneck. This research provides a critical empirical roadmap for practitioners and researchers, demonstrating that while prompt optimization is a viable path to performance improvement, it requires a principled, configuration-aware approach to avoid unintended performance degradation.