MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving

Autonomous driving systems often struggle with multi-task learning, where the performance of joint models (e.g., simultaneous 3D object detection and map segmentation) is lower than that of independent, task-specific models—a phenomenon known as negative transfer. Existing methods typically compress multi-modal sensor data into a single, shared bird's-eye-view (BEV) feature map, which often fails to capture the distinct, task-specific information required for high-accuracy perception.

The authors propose MATS, a framework designed to mitigate negative transfer by replacing the single shared BEV map with a more flexible architecture. The approach consists of two primary innovations:

• Modality-Adaptive BEV Fusion: Instead of a single fusion strategy, the model uses multiple independent fusion branches. Each branch adaptively recalibrates camera and LiDAR features to generate diverse BEV feature maps, ensuring that different task-relevant information is preserved.
• Task-Specific Mixture-of-Experts (MoE): To prevent task conflict, the model employs an MoE module that routes features to specialized experts based on the task type (detection vs. segmentation). This decouples the optimization targets, allowing the network to select the most appropriate BEV feature candidate for each specific task.

MATS was evaluated on the large-scale nuScenes benchmark. The experimental results demonstrate that the framework significantly outperforms state-of-the-art methods in joint 3D object detection and map segmentation. Furthermore, the model shows superior performance even when evaluated on individual tasks compared to standard baselines, confirming that the proposed fusion and MoE strategies effectively reduce negative transfer and improve overall perception accuracy.

By enabling effective multi-task learning, MATS reduces the computational overhead and deployment complexity associated with running multiple independent perception models. This makes it a more practical solution for real-world autonomous driving systems that require high-performance, real-time environmental understanding.