Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions

Large language model (LLM) agents are increasingly used to simulate human behavior in complex systems, but their generative nature poses a validity problem: agents may produce individually plausible narratives that fail to match the statistical structure of real-world population behavior. This study investigates whether "empirical grounding"—integrating real-world demographic and time-use data into agent initialization and decision-making—can improve the statistical realism of these simulations during infrastructure disruptions like heatwaves.

The authors developed an empirically grounded LLM-agent framework that incorporates data from the American Community Survey (demographics) and the American Time Use Survey (baseline routines), alongside urban spatial constraints. They tested this framework against an ungrounded baseline using a July 2024 Philadelphia heatwave as a case study. The model's performance was validated against an independent household survey conducted during the same heatwave, focusing on the agents' ability to reconstruct normal daily routines and simulate adaptive behavioral shifts under heat stress.

Empirical grounding dramatically improved the statistical accuracy of the simulations. Under normal conditions, the grounded model increased the mean correlation with empirical activity profiles from 0.528 to 0.912. During the heatwave, the grounded model significantly outperformed the ungrounded baseline, increasing the correlation with observed activity profiles from 0.349 to 0.836 and more than doubling the captured share of observed behavioral adaptation (from 20.6% to 46.4%). While the grounded model successfully captured the general direction of behavioral changes, it still underestimated the full magnitude of adaptation, highlighting the need for more sophisticated modeling of institutional and household constraints.