Search-based Testing of Vision Language Models for In-Car Scene Understanding

In-car scene understanding (ISU) systems, which use vision-language models (VLMs) to interpret cabin activity, are critical for safety features like driver distraction detection. However, these systems are prone to errors, and collecting real-world data for testing is costly and difficult to scale. The authors introduce ISU-Test, an automated testing framework that treats the generation of in-car scenarios as an optimization problem. By combining a rendering engine with search-based testing, the framework systematically modifies scene parameters—such as driver pose, object placement, and lighting—to identify scenarios where the VLM provides incorrect or incomplete descriptions.

The ISU-Test framework operates by defining a search space of scene features (e.g., driver emotion, seatbelt status, luggage presence) and using a genetic algorithm to iteratively generate and mutate scenes. The system evaluates these scenes using two primary modes: Visual Question Answering (VQA), which uses structured fitness functions to compare model outputs against ground truth, and Visual Captioning (VC), which uses semantic similarity metrics like BERTScore and BLEU to evaluate open-ended descriptions. A key innovation is the use of both response-oriented fitness functions (to find failures) and diversity-oriented fitness functions (to ensure a wide variety of failure modes are explored).

ISU-Test significantly outperforms random scenario generation across both VQA and VC tasks. In VQA, the framework achieved up to 10 times higher failure rates, while in VC, it demonstrated up to 3.6 times higher failure coverage. The study also highlights that ISU-Test explores a more balanced distribution of failure modes, as evidenced by higher entropy scores in failure clustering. Furthermore, a validation study comparing simulated scenes to real-world reconstructions showed an 89% agreement rate, confirming that the framework serves as a reliable proxy for early-stage system validation.