Haibo Wang, Lifu Huang
7 min
Abstract
Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames. Given the scarcity of large-scale 3D data, we present GeoVR, a novel framework that learns geometric representations using purely 2D video sequences. This approach effectively restructures the semantic latent space within MLLMs to unlock spatial intelligence. Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge from pre-trained 3D foundation models. This is accomplished through a multi-objective learning strategy driven by four complementary geometric targets: (1) estimating inter-frame camera poses to embed varying viewpoint dynamics, (2) regressing dense depth maps to anchor physical distances, (3) predicting a metric scale factor for real-world calibration, and (4) distilling multi-scale 3D features to align the intermediate feature space. Guided by these explicit physical and geometric constraints, the model's internal representations naturally develop strong 3D awareness. Extensive experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance, establishing a new paradigm for endowing foundation models with spatial intelligence.
Sam: Correct. All that geometric knowledge gets absorbed into the model's internal representations during training. After that, the teacher model is discarded entirely. The AI is left with a native sense of 3D space—no external sensor running in the background, no extra cost during actual use.
Alex: That's a significant efficiency gain. But how does it handle the real world, where things aren't always cleanly measured?
Sam: That's where a specific piece of the design becomes important. In the real world, AI models often struggle with something researchers call scale ambiguity. Imagine looking at a photo of a ball—you can't tell if it's a tennis ball one metre away or a beach ball twenty metres away. To address this, the researchers added something they call a scale token. Think of it as a dedicated internal sensor that looks at the entire video and calculates the true physical size of the environment—like giving the AI a mental ruler it can use to calibrate its own perception.
Alex: And that calibration is built into the training, not bolted on afterward?
Sam: Exactly. The model learns to output a single number representing the absolute scale of the scene, and the researchers train this in a way that keeps the system stable across vastly different environments—whether it's a small kitchen or a large warehouse.
Alex: Beyond scale, is there another layer to how the AI aligns its thinking with the teacher model?
Sam: There is, and it's arguably the most important part. Beyond predicting numbers like depth or scale, the researchers force the AI to align its internal representations—its actual thought patterns—with those of the teacher model, layer by layer. Imagine a student learning to paint by comparing not just the finished canvas, but each individual brushstroke, to a master's work. The goal is to ensure the student is capturing the same sense of depth and perspective at every stage of the process, not just getting the final picture to look right.
Alex: So it's checking the AI's reasoning at every internal step, not just the final answer?
Sam: Correct. Because the AI and the teacher have different internal structures, the researchers use a bridging function to make them comparable—essentially resizing and reformatting the AI's internal data so it can be measured against the teacher's. They then check how similar the two patterns are. If they match closely, the AI has successfully absorbed the teacher's 3D intuition.
Alex: And all of this—language understanding, depth, camera movement, scale—is learned simultaneously?
Sam: Yes. The model is trained on all of those objectives at once, which is what makes the approach coherent rather than piecemeal. And because the teacher components are only needed during training, the final deployed model carries none of that extra weight. It's geometry-aware from the inside out.
Alex: How do the researchers actually verify that the internal restructuring worked? You can't exactly look inside an AI's brain.
Sam: They used a technique called feature visualization. The idea is to take the AI's internal data—which is far too complex to look at directly—and project it into a colour map that represents spatial layout. In the original model, those maps look like noisy, chaotic patterns. After GeoVR training, they become smooth and structured, with clear boundaries between objects, almost like a proper depth map of the scene.
Alex: So you can literally see the difference in how the model is organizing information?
Sam: You can. And they went further—they took the model's predicted depth and camera movement data and used it to reconstruct what's called a 3D point cloud from a standard video. A point cloud is essentially a collection of dots in 3D space that together represent the shape of a room or object. The results show the model can recover the structure of a real environment with meaningful accuracy, which suggests it has genuinely internalized the physical layout rather than just pattern-matching on surface features.
Alex: Are there limits to what GeoVR can do?
Sam: There are, and the researchers are clear about them. The framework is only as good as the teacher model it learns from. If the teacher has blind spots or errors in its understanding of 3D space, the student AI inherits those same weaknesses. It can't exceed the quality of the geometric knowledge it's fed during training—like a student who can only be as accurate as their textbook.
Alex: That's an important caveat.
Sam: They also found that the alignment process has to be done carefully. When they tried matching only certain internal layers rather than all of them, performance dropped noticeably. Teaching an AI to think in 3D requires a balanced, multi-level approach—you can't just focus on the surface and ignore what's happening deeper in the network.
Alex: Stepping back—why does this matter beyond the lab?
Sam: The long-term implication is about accessibility. Right now, building systems with reliable 3D perception typically requires expensive specialized hardware—laser scanners, depth cameras, carefully controlled environments. If you can train a model to develop spatial intelligence from ordinary video footage that already exists in enormous quantities, you lower that barrier considerably. Navigation systems, robotics, assistive technology—these could all benefit from models that understand physical space without needing a dedicated 3D sensor on every device.
Alex: It's a meaningful shift in how we think about what a model needs to learn, versus what it needs to carry around with it.
Sam: That's a good way to frame it. The research suggests that spatial intelligence doesn't have to be an expensive add-on. With the right training structure, it can become part of how a model thinks from the ground up. Thanks for listening to ResearchPod.