HETEROPROMPT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework

Collaborative Perception (CP) allows autonomous vehicles to share sensor data to improve environmental awareness. However, real-world deployments often involve heterogeneous agents—vehicles with different sensors, perception models, and training domains. This heterogeneity creates feature-space shifts that degrade fusion and detection performance. The authors seek a way to align these diverse features in real-time while preserving agent privacy and avoiding the high computational costs of retraining entire perception stacks.

The authors propose HeteroPROMPT, a framework that uses modular prompts and learning-based tuning to align incoming features to an ego-centric unified space. The system operates in two main stages:

1. Unified Space Learning: The system first establishes a homogeneous base using a pyramid fusion architecture.
2. DRIFT Adaptation: To handle new modalities, the authors introduce the Decomposed-Rank Interpreter for FiLM Tuning (DRIFT). This module uses visual prompts and Feature-wise Linear Modulation (FiLM) to adaptively modulate Bird's Eye View (BEV) features. To maintain efficiency, the authors apply PARAFAC low-rank decomposition to these prompts, significantly reducing the number of trainable parameters.

To enable deployment without sharing proprietary metadata, the framework includes an AutoEncoder (AE) trained to map features into a compact latent space. A lightweight classifier then identifies the incoming agent's modality in real-time, routing the data to the correct DRIFT module.

Experiments on the OPV2V-H and V2XSet datasets demonstrate that HeteroPROMPT achieves state-of-the-art detection accuracy while reducing training overhead by 96% compared to existing methods. The modality classification scheme is highly effective, identifying the joining agent's modality with over 99.99% accuracy. By keeping the main fusion and detection stacks frozen, the framework provides a scalable, privacy-preserving solution that avoids the need to retrain models when new agents join the network.