TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

Existing cross-modal distillation methods for LiDAR often treat Vision Foundation Models (VFMs) as frozen feature sources and attempt to force a heterogeneous 3D backbone to match fixed image embeddings. This approach forces the student model to bridge both a modality gap and a significant architectural gap between dense 2D tokens and sparse 3D point features, often leading to information loss and unstable transfer. The authors ask whether aligning LiDAR features with ViT-compatible tokens—rather than direct feature regression—can improve the quality and transferability of 3D representations.

To address these gaps, the authors propose TOLiD, a self-supervised pretraining framework. Instead of distilling directly into a 3D backbone, TOLiD couples the LiDAR encoder with a student ViT initialized from a frozen VFM teacher. The method uses two key mechanisms to bridge the modality gap:

• Frustum Pooling (FP): Maps variable-sized point sets within an image patch frustum into a fixed-dimensional token at the input stage.
• Frustum Attention (FA): A cross-attention mechanism in deeper ViT blocks that allows tokens to selectively aggregate informative 3D points within their corresponding frustum, effectively injecting geometric structure into the semantic processing pipeline.

During pretraining, the model performs visibility-masked distillation, ensuring that supervision is applied only to patches supported by actual LiDAR geometry. For deployment, the authors use Masked Bilinear Sampling (MBS) to lift the learned token features back into the 3D point domain, allowing the model to be used for downstream tasks like semantic segmentation without requiring the image-based student ViT.

Evaluated across five heterogeneous LiDAR datasets and four cross-sensor adaptation pairs, TOLiD demonstrates superior performance compared to existing distillation pipelines. The authors observe that TOLiD scales effectively with pretraining diversity; as the amount of pretraining data increased from 169k to 342k frames, the performance gap over the state-of-the-art widened from +2.0% to +4.2% mIoU. The resulting representations are highly robust, achieving state-of-the-art accuracy on target domains even when using a frozen backbone and a simple, lightweight MLP head.