Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records

Testing autonomous driving systems (ADS) for safety is a critical but costly challenge. Traditional methods rely on manually designed, static scenario templates that often fail to capture the nuance of real-world accidents. This paper investigates how to automate the generation of diverse, system-specific test scenarios by leveraging historical crash records, specifically using natural language narratives to inform simulation parameters.

The researchers developed a three-stage pipeline that processes NHTSA crash records to generate synthetic test scenarios. First, they extract categorical meta-variables (e.g., road type, vehicle movement) and qualitative narratives from crash reports. Second, they employ a modular LLM-based framework consisting of three agents: a paraphrasing agent to refine narratives for simulation compatibility, a scenario generation agent to translate these into simulator-specific parameters, and a fine-tuning agent to optimize scenarios based on performance metrics like minimum gap between vehicles. Finally, they use k-means clustering to select a diverse set of 20 scenarios for testing an Intelligent Driver Model (IDM) policy in the Metadrive simulator.

The pipeline successfully generated a diverse test suite that captured 11 out of 14 scenario templates found in the historical data. The generated scenarios revealed specific system weaknesses, such as oscillatory movement during straight-path navigation and side-impact collisions during lateral maneuvers. The fine-tuning agent proved effective, significantly reducing the minimum gap between vehicles in 14 out of 16 tested cases, thereby increasing the likelihood of exposing failure modes within a constrained testing budget.

This work bridges the gap between static, expert-designed test suites and adaptive, simulation-based testing. By automating the extraction of failure conditions from real-world data, the proposed method reduces the manual effort required for safety validation and allows developers to stress-test specific ADS policies against both common and rare failure conditions. This approach is modular and can be integrated with existing optimization tools like Bayesian optimization or MCMC sampling to further explore the failure space.