SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision

Multimodal Large Language Models (MLLMs) often struggle with consistent 3D spatial reasoning because they are primarily trained on 2D visual-textual data. While some methods attempt to fix this by distilling features from 3D models or using external tools, these approaches often lack interpretability and impose significant inference overhead. This paper asks: can we force MLLMs to internalize 3D spatial awareness in a way that is both robust and inherently interpretable?

The authors introduce SpatialSV, a framework that moves beyond passive feature distillation. Instead, it employs task-oriented visual supervision. The model is trained to actively "lift" its internal 2D visual features into explicit 3D representations, specifically depth maps, camera poses, and point clouds. By using a set of decoupled DPT (Dense Prediction Transformer) modules, the model learns to predict these geometric properties during training. This process acts as a "transparent window" into the model's internal state, allowing researchers to visualize and diagnose the quality of the model's spatial knowledge by inspecting the generated 3D outputs.

SpatialSV consistently outperforms standard text-only fine-tuning and traditional feature distillation across multiple MLLM architectures and benchmarks. The authors demonstrate that the quality of the model's internal 3D representations is strongly correlated with its spatial reasoning performance. Furthermore, the framework shows strong generalization in semi-supervised settings, proving that it can effectively learn spatial representations even when textual annotations are scarce. Because the model is forced to generate explicit 3D outputs, researchers can intuitively assess the model's spatial understanding by examining the quality of these generated maps.

This work addresses a fundamental bottleneck in embodied AI and robotics: the gap between 2D visual processing and 3D world understanding. By internalizing spatial awareness, MLLMs can potentially perform complex spatial reasoning tasks without needing to rely on heavy, external 3D tools during inference. This makes the models more efficient and provides a much-needed mechanism for interpreting how these "black-box" models perceive the geometry of the world around them.