MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers

As Model Context Protocol (MCP) servers become the standard for connecting LLMs to external tools, existing benchmarks have largely treated these tool environments as static. However, real-world software is dynamic, with APIs and tool interfaces undergoing constant updates, deprecations, and functional changes. This paper introduces MCPEvol-Bench to evaluate how well LLM agents adapt to these evolving tool landscapes, addressing the critical gap where agents fail to maintain workflow integrity when their underlying tools change.

To create a realistic evaluation environment, the authors first conducted a large-scale empirical study of 515 MCP servers, identifying 11 distinct mutation operators—ranging from parameter adjustments to tool deletions—that reflect how developers actually update software. Using these operators, the researchers implemented an automated pipeline where LLMs iteratively modify the source code of 123 MCP servers. This process generated a multi-version benchmark of 201 complex tasks, allowing for the systematic assessment of 12 state-of-the-art LLMs across different stages of tool evolution.

Evaluation results reveal that even the most capable frontier models struggle significantly as tool environments evolve. Performance drops are not merely due to simple tool failures but are driven by a 34.1% increase in planning errors and a 35.6% increase in reasoning errors. The study highlights that while removing redundant tools has little impact, modifications to tool interfaces or the addition of new tools severely disrupt agent performance. The authors conclude that current agent architectures lack the necessary cognitive robustness to handle environmental drift, suggesting that future systems must incorporate better reflection and memory modules to maintain adaptability.