Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how companies test self-driving cars. It sounds like they're currently flying a bit blind, right?
Sam: That's a fair assessment. This paper examines Autonomous Driving Systems—or ADS—the full set of software and hardware that allows a vehicle to drive itself. The central puzzle is that while these systems are advancing, the industry lacks a unified, standard way to prove they are actually safe.
Alex: So this paper is asking how we move from messy, random testing to a more reliable, professional standard?
Sam: Exactly. The researchers interviewed experts from nine different companies to understand why testing is so difficult. They found that most teams use a mix of two approaches: "scenario-based testing," where you run the car through specific, pre-planned events like a sudden stop, and something called "X-in-the-loop" testing.
Alex: What does "X-in-the-loop" actually mean? That sounds like jargon.
Sam: Think of it like a flight simulator for a pilot. Before you put the software on a real road, you plug the computer brain into a virtual environment—or hook it up to real steering parts in a lab—to see how it performs. You are putting the system "in the loop" of the test. It's just a way of saying: let's test this piece of the car by making it think it's driving, even if it's sitting in a room.
Alex: That makes sense. But if they're already doing that, why is it still so hard to know if the car is safe?
Sam: The core problem is that companies are often measuring the wrong things. Many rely on counting total miles driven, but quantity isn't the same as quality. You could drive a million miles on a straight, empty highway and learn very little about how the car handles a complex, rainy intersection. The researchers argue we need to stop just collecting data and start anchoring tests in specific, evidence-based safety claims.
Alex: So instead of just driving more, they need to prove the car can handle specific, dangerous situations?
Sam: Precisely. They propose what they call an "evidence-centered closed-loop framework." Think of it like a doctor diagnosing a patient. You don't run every possible scan—you identify a specific symptom, a safety claim, and then pick the exact test needed to check for that problem. If the test fails, you use that information to improve the system and add that situation to your list of scenarios. It creates a cycle where the testing gets smarter over time.
Alex: So the car's failures directly help build the next, better set of tests. But how do they handle the gap between a perfect simulation and the messy real world?
Sam: That's a real tension. If a simulator is too simple, the car might develop habits that don't hold up on actual pavement. Teams are now trying to match the simulation's level of detail to the specific task—using high-quality, realistic graphics when testing the camera system, but simpler models when testing the steering response. The idea is to choose the right tool for each job.
Alex: Is there a standard way to decide which scenarios are most important to test in the first place?
Sam: That's the central challenge. Because the number of possible driving situations is effectively infinite, teams have to prioritize. They use a mix of regulatory requirements, industry standards, and data from real-world driving to build their library. They call this "scenario coverage"—it's really just asking: have we tested enough variety to be confident?
Alex: But "variety" is vague. What about genuinely unexpected things—like a person wearing a costume, or a strangely shaped obstacle the car has never seen before?
Sam: That's the primary concern. Participants in the study noted that systems often perform well on standard objects—cars, traffic cones—but struggle with things outside their normal training. One example mentioned was a black pig dummy placed on a highway. A safe system needs to handle these arbitrary obstacles, not just the familiar ones. The researchers call these "out-of-distribution" items, meaning things the system simply wasn't prepared for.
Alex: So if they can't predict every possible object, how do they decide when testing is actually done? Is there a number of miles that proves it's safe?
Sam: The experts were clear: mileage is a misleading metric. Instead, they're shifting toward what you might call behavioral consistency—they keep testing until the vehicle responds the same way to a given situation every single time, across a wide range of scenarios. That predictability is what builds genuine confidence.
Alex: And when something does go wrong after the car is deployed on real roads?
Sam: They don't just move on. They feed that specific failure back into their system—they call it "log replay"—and use automated tools to label the data and run that exact situation through their simulators again to make sure the fix actually works. And then they rerun all their previous tests to make sure the new fix didn't accidentally break something else that was already working. It's a daily cycle: identify a failure, update the software, re-test everything.
Alex: That sounds like a rigorous, never-ending process.
Sam: It is. And that's actually the point. This framework isn't about finding perfection—it's about building a discipline that can reliably identify and fix weaknesses as they appear. The shift is from ad-hoc testing, where you just drive and hope, to a systematic approach where every test serves a specific purpose.
Alex: What are the limitations? It sounds almost too tidy.
Sam: The biggest limitation is that this study is based on qualitative interviews with experts. It provides a useful conceptual map, but it doesn't offer a single universal metric that spits out a "safe" or "unsafe" verdict. It's a framework for organizing how you think about the problem—not a shortcut to the answer. And it doesn't fully solve the sim-to-real gap either. It just gives teams a structured way to acknowledge those gaps and plan around them.
Alex: So it's a guide for how to think, not a formula.
Sam: Exactly. And when you zoom out, that shift in thinking matters. The field is moving toward a future where testing is continuous and self-correcting—where the car's own failures in the real world feed directly back into making it safer. It's a more honest way of approaching a genuinely difficult problem.
Alex: Rather than pretending you can test everything in advance, you build a system that keeps learning from what it gets wrong.
Sam: That's it. And the next challenge for the field will be improving how efficiently that learning happens—getting more insight from fewer tests, rather than just running more miles. That's where the research is likely to go next.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.