CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving

Autonomous vehicles rely on sensor fusion—combining data from cameras, LiDAR, and radar—to perceive their environment. However, sensor reliability fluctuates significantly due to environmental factors like lighting, weather, or sensor noise. Traditional uncertainty quantification (UQ) methods often struggle to generalize to out-of-distribution (OOD) scenarios because they rely on coarse, feature-level uncertainty estimates that fail to capture spatial nuances. The authors propose CRUISE, a framework that integrates a Vision-Language Model (VLM) to provide fine-grained, pixel-level uncertainty guidance for sensor fusion.

CRUISE operates in two primary stages. First, it uses a VLM-guided UQ module. By comparing test samples against a stored 'feature pool' of representative training data, the system generates initial uncertainty estimates. These are then refined by a VLM (specifically, a fine-tuned AnomalyGPT) that leverages its pre-trained contextual reasoning to produce precise uncertainty heatmaps. Second, the framework employs a dynamic adaptation mechanism that explicitly models cross-modal dependencies, allowing the system to adaptively weigh sensor inputs based on the VLM’s uncertainty maps.

Experimental results demonstrate that CRUISE significantly outperforms state-of-the-art fusion methods. The authors report an average accuracy improvement of 4.87% in 3D object detection and 4.23% in semantic segmentation. Furthermore, the VLM-guided UQ module shows superior generalization to complex, OOD driving scenarios, outperforming traditional UQ techniques by an average of 4.16%. The study highlights that by moving from coarse, modality-level uncertainty to fine-grained, spatial uncertainty, the system can more effectively suppress misleading signals and exploit the complementary strengths of different sensors.

This research addresses a critical bottleneck in autonomous driving: the inability of perception systems to reliably handle unexpected or adverse conditions. By leveraging the rich prior knowledge inherent in large-scale vision-language models, CRUISE provides a more robust way to quantify uncertainty in real-time. This approach not only improves detection accuracy but also enhances the safety and dependability of autonomous systems in unpredictable real-world environments.