HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training

Autonomous driving systems require robust 3D perception, but training LiDAR models typically demands massive, manually labeled datasets. While cross-modal knowledge distillation from Vision Foundation Models (VFMs) has emerged as a solution, existing methods often treat the teacher as a black box, distilling only final-layer features and ignoring the rich spatiotemporal structure of LiDAR sequences. This paper asks: can we improve LiDAR representation learning by better exploiting the hierarchical semantic structure of VFMs and explicitly modeling 3D geometric dynamics?

The authors propose HilDA, a framework that introduces three key components to the pre-training process:

• Multi-layer Distillation: Instead of only using the final layer of the VFM, the model distills features from multiple intermediate layers to capture the progressive semantic refinement inherent in transformer-based vision encoders.
• Global Context Distillation: The model aligns the VFM's global [CLS] token with a learnable 3D global context token from the LiDAR student, ensuring the 3D backbone captures holistic scene-level semantics.
• Temporal Occupancy Diffusion: The authors introduce an auxiliary generative task where the model learns to predict future BEV (Bird's Eye View) occupancy using a conditional diffusion model. This forces the student to internalize spatiotemporal consistency and geometric structure.

HilDA consistently outperforms existing state-of-the-art distillation methods across a wide range of autonomous driving benchmarks, including 3D object detection, semantic occupancy prediction, and scene flow. The improvements are particularly pronounced in data-scarce scenarios (e.g., 1%–10% label availability), demonstrating that the hierarchical and generative objectives produce more transferable representations than standard final-layer distillation. The authors show that each component—multi-layer alignment, global context, and diffusion—contributes incrementally to reducing semantic segmentation errors.