Decision Protocols in Multi-Agent Large Language Model Conversations

This study investigates how different decision-making protocols influence the performance of Multi-Agent Systems (MAS) powered by Large Language Models (LLMs). As scaling models faces diminishing returns and high costs, distributing tasks among specialized agents offers a way to improve performance. The author introduces the Multi-Agent LLM (MALLM) framework to systematically evaluate three primary decision mechanisms—voting, consensus, and judge-based protocols—across a diverse range of knowledge-based and logic-based datasets.

The MALLM framework simulates multi-agent discussions where agents collaborate to solve conversational tasks. The research compares three distinct protocols:

• Voting: Agents cast votes on proposed solutions.
• Consensus: Agents must collaboratively agree on a final solution.
• Judge: A designated agent evaluates and selects the best solution from the pool.

The study tests these protocols on datasets including MMLU, GPQA, StrategyQA, and Math-lvl-5. By keeping the underlying model structure consistent, the author ensures that performance differences are attributable to the decision protocols rather than model variations. The experiments also explore how response diversity, the number of agents, and information access during the decision phase affect final outcomes.

The results demonstrate that there is no single "best" protocol for all scenarios. Consensus protocols excel in knowledge-intensive domains, likely due to the requirement for agreement, while voting and judge protocols are better suited for logic-based tasks where individual reasoning paths can be evaluated. The study also finds that increasing the diversity of initial solutions—rather than just iterating on a single path—significantly improves the quality of the final decision. Interestingly, while the number of agents involved can boost performance, excessively long discussions lead to diminishing returns and increased computational overhead.