Automated Generation of Complexity-Validated Decision Scenarios Using Large Language Models

This study addresses the challenge of creating standardized, complexity-controlled decision scenarios for cognitive research. Manual scenario generation is often slow and prone to bias, hindering the scalability of cognitive assessments. The authors developed an automated pipeline that uses Large Language Models (LLMs) to generate structured scenarios based on a 14-field JSON schema. To ensure scientific rigor, they implemented a multi-framework complexity scoring system derived from established theories by Wood, Campbell, Liu and Li, and Sweller. This system evaluates scenarios across four dimensions: component/coordinative complexity, decision-path enumeration, information processing demands, and element interactivity.

The researchers validated the system using 4,238 generated scenarios. Psychometric testing confirmed strong construct validity, with a dominant structural factor explaining the majority of the variance. Inter-model agreement was exceptionally high, with an intraclass correlation coefficient (ICC) of 0.997, indicating that different LLM families consistently apply the same complexity standards. Known-groups validity testing showed clear, statistically significant separation between the intended complexity tiers (Simple, Moderate, and Complex).

The study also evaluated the performance of different LLMs in generating these scenarios. While all models could produce valid outputs, there were notable differences in throughput and quality. A speed-quality trade-off was observed, where high-throughput models sometimes struggled to maintain the schema compliance or domain coverage required for complex-tier scenarios. For instance, while some models were significantly faster, others like DeepSeek Chat V3.2 proved more effective at balancing domain coverage with high schema compliance.