Towards Agentic Agent-based Models: Feasibility, Performance, and Statistical Model Checking

This paper investigates the methodological implications of replacing traditional, rule-based decision-making in agent-based models (ABMs) with agentic capabilities powered by Large Language Models (LLMs). The authors seek to understand whether LLMs can reliably perform local agent tasks, such as neighbor classification, and how their integration affects the computational performance and emergent behavior of the simulation.

To study this, the researchers use the Mesa framework to implement a hybrid version of the classic Schelling segregation model. In this version, one agent delegates its neighbor-classification task to a locally served LLM via tool calls, while the rest of the population follows standard symbolic rules. The authors utilize Statistical Model Checking (SMC) through the MultiVeStA tool to quantify the impact of these LLM-driven decisions on the system's transient behavior and to compare the performance of different LLM sizes.

The study highlights a clear trade-off between model size, reliability, and computational cost. Smaller LLMs (e.g., 0.8b parameters) frequently failed to perform basic semantic classification or struggled to generate valid tool calls, rendering them unsuitable for the simulation loop. Larger models were more robust but imposed a significant runtime overhead on the simulation. Despite these operational differences, the authors found that the overall emergent dynamics of the Schelling model—specifically the ratio of happy agents over time—remained consistent with the original symbolic model, suggesting that the LLM-based agent successfully replicated the intended logic despite the added complexity.

As researchers increasingly look to LLMs to create more "agentic" and realistic simulations, this paper provides a necessary methodological framework for validating these models. By demonstrating that LLM-enabled agents can introduce non-trivial operational failures and performance bottlenecks, the authors argue that such agents should not be treated as drop-in replacements for symbolic rules. Instead, they advocate for the use of statistical model checking to rigorously quantify the impact of LLM integration, ensuring that the resulting simulations remain reproducible and scientifically valid.