Towards Collaborative Joint Perception and Prediction: Framework, Baseline Evaluation, and Deployment Perspectives

As Connected Autonomous Vehicles (CAVs) become more common, they face persistent challenges like visual occlusions and the accumulation of errors in modular perception pipelines. This paper explores whether unifying collaborative perception (CP) with motion prediction—a paradigm termed Collaborative Joint Perception and Prediction (Co-P&P)—can enhance situational awareness and prediction reliability in complex traffic environments.

The authors propose a conceptual Co-P&P framework that separates collaborative scene representation from downstream perception tasks. The system integrates three main components: a Collaborative Scene Completion (CSC) module that aggregates LiDAR data from multiple agents, a localization module that aligns these inputs, and a Joint Perception and Prediction (P&P) module. To test this, the researchers implemented an end-to-end prototype using the RENO neural codec for bandwidth-efficient point-cloud sharing and FutureDet for joint detection and forecasting. They also conducted a systematic comparison of fusion strategies, testing whether it is more effective to fuse data at the detection, tracking, or prediction stage.

The experimental results demonstrate that collaborative sharing significantly improves forecasting accuracy compared to single-vehicle baselines. A key technical finding is that the timing of fusion matters: prediction-level fusion actually degrades system performance compared to earlier fusion stages (detection or tracking). Furthermore, the authors show that using neural compression (RENO) to share intermediate features allows the system to achieve these performance benefits while reducing the required communication bandwidth by approximately 34 times.

This work provides a reproducible baseline for collaborative perception and prediction, moving beyond simple object-list sharing toward more sophisticated, bandwidth-efficient feature sharing. By addressing the cumulative error problem inherent in traditional modular pipelines, the Co-P&P framework offers a scalable path toward safer autonomous navigation in occluded or high-density traffic scenarios.