ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models

In Vehicle-to-Everything (V2X) collaborative perception, LiDAR backbones are typically trained from scratch, resulting in geometry-dominated features that lack general semantic knowledge. This paper investigates whether Vision Foundation Models (VFMs), which possess strong general-purpose visual representations, can be adapted to enhance LiDAR-based BEV features to improve collaborative 3D object detection performance.

ViCo3D introduces a dual-branch framework that processes LiDAR data through two parallel streams. The first branch uses a standard PointPillars encoder to capture geometric features. The second branch projects point clouds into three-channel BEV images (height, intensity, and density) and processes them using DINOv2 to extract semantic visual features. To bridge the modality gap between these two streams, the authors implement a multi-scale fusion module consisting of:

• Low-Resolution Global Fusion (LRG): Uses transformer-based attention on downsampled features to capture long-range spatial dependencies.
• High-Resolution Refinement Fusion (HRR): Uses lightweight convolutions on original-resolution features to preserve fine-grained object boundaries and local spatial details.

Finally, the enhanced features are aggregated across agents using an ego-centric cross-agent fusion strategy, which employs a gating mechanism to adaptively weight complementary information from neighboring vehicles and infrastructure.

ViCo3D achieves state-of-the-art performance on the DAIR-V2X and V2XSet datasets. On DAIR-V2X, the model reaches 82.80 and 71.39 AP at IoU thresholds of 0.5 and 0.7, respectively. Notably, the framework demonstrates superior collaborative gains, delivering up to 1.8x greater improvement over single-vehicle baselines compared to previous state-of-the-art methods like INSTINCT and DI-V2X. The results suggest that integrating VFM-derived visual priors significantly mitigates the limitations of geometry-only LiDAR representations in multi-agent environments.