Qunying Song, Yuan Gao, Johannes Betz, Dietmar Pfahl, Mohammad Reza Mousavi, Federica Sarro
6 min
Abstract
Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.
Sam: That's a real tension. If a simulator is too simple, the car might develop habits that don't hold up on actual pavement. Teams are now trying to match the simulation's level of detail to the specific task—using high-quality, realistic graphics when testing the camera system, but simpler models when testing the steering response. The idea is to choose the right tool for each job.
Alex: Is there a standard way to decide which scenarios are most important to test in the first place?
Sam: That's the central challenge. Because the number of possible driving situations is effectively infinite, teams have to prioritize. They use a mix of regulatory requirements, industry standards, and data from real-world driving to build their library. They call this "scenario coverage"—it's really just asking: have we tested enough variety to be confident?
Alex: But "variety" is vague. What about genuinely unexpected things—like a person wearing a costume, or a strangely shaped obstacle the car has never seen before?
Sam: That's the primary concern. Participants in the study noted that systems often perform well on standard objects—cars, traffic cones—but struggle with things outside their normal training. One example mentioned was a black pig dummy placed on a highway. A safe system needs to handle these arbitrary obstacles, not just the familiar ones. The researchers call these "out-of-distribution" items, meaning things the system simply wasn't prepared for.
Alex: So if they can't predict every possible object, how do they decide when testing is actually done? Is there a number of miles that proves it's safe?
Sam: The experts were clear: mileage is a misleading metric. Instead, they're shifting toward what you might call behavioral consistency—they keep testing until the vehicle responds the same way to a given situation every single time, across a wide range of scenarios. That predictability is what builds genuine confidence.
Alex: And when something does go wrong after the car is deployed on real roads?
Sam: They don't just move on. They feed that specific failure back into their system—they call it "log replay"—and use automated tools to label the data and run that exact situation through their simulators again to make sure the fix actually works. And then they rerun all their previous tests to make sure the new fix didn't accidentally break something else that was already working. It's a daily cycle: identify a failure, update the software, re-test everything.
Alex: That sounds like a rigorous, never-ending process.
Sam: It is. And that's actually the point. This framework isn't about finding perfection—it's about building a discipline that can reliably identify and fix weaknesses as they appear. The shift is from ad-hoc testing, where you just drive and hope, to a systematic approach where every test serves a specific purpose.
Alex: What are the limitations? It sounds almost too tidy.
Sam: The biggest limitation is that this study is based on qualitative interviews with experts. It provides a useful conceptual map, but it doesn't offer a single universal metric that spits out a "safe" or "unsafe" verdict. It's a framework for organizing how you think about the problem—not a shortcut to the answer. And it doesn't fully solve the sim-to-real gap either. It just gives teams a structured way to acknowledge those gaps and plan around them.
Alex: So it's a guide for how to think, not a formula.
Sam: Exactly. And when you zoom out, that shift in thinking matters. The field is moving toward a future where testing is continuous and self-correcting—where the car's own failures in the real world feed directly back into making it safer. It's a more honest way of approaching a genuinely difficult problem.
Alex: Rather than pretending you can test everything in advance, you build a system that keeps learning from what it gets wrong.
Sam: That's it. And the next challenge for the field will be improving how efficiently that learning happens—getting more insight from fewer tests, rather than just running more miles. That's where the research is likely to go next.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.