Boosting Text-Driven Video Segmentation via Geometry-Aware Distillation

Text-driven Referring Video Object Segmentation (RVOS) often struggles with spatial understanding and temporal consistency because models are typically trained on static 2D images. The authors investigate whether explicitly integrating 3D geometric priors—both through synthetic multi-view data and distillation from 3D foundation models—can improve the model's ability to track and segment objects in dynamic video scenes.

The authors propose GeoLaV, a two-stage framework designed to inject 3D awareness into a video segmentation backbone (based on SAM2).

• Stage I (Monocular Geometry Pretraining): The model is trained on synthetic video sequences generated from single images. By using monocular depth estimation and novel-view synthesis, the authors create sequences with physically valid camera motion, forcing the model to learn geometry-consistent visual representations.
• Stage II (Geometry-Aware Distillation): During fine-tuning on real video datasets, the model uses a distillation process to align its internal features with those of frozen 3D-aware foundation models (e.g., π3). This forces the segmentation backbone to incorporate 3D structural and motion cues, improving its robustness to complex camera movements.

GeoLaV demonstrates that incorporating geometric priors significantly enhances performance in RVOS tasks. The monocular pretraining stage alone provides notable zero-shot generalization capabilities, even when the model has not yet seen real video data. When combined with 3D-aware distillation, the framework achieves state-of-the-art results across multiple standard RVOS benchmarks, showing superior stability and tracking accuracy compared to methods that rely solely on 2D image-based pretraining.

This work highlights that the primary bottleneck in current video segmentation models is not just the lack of video data, but the lack of 3D geometric reasoning. By leveraging synthetic data and 3D priors, the authors provide a scalable way to improve video understanding without requiring massive amounts of expensive, manually annotated video data.