Vision-language models (VLMs) achieve strong performance on spatial reasoning benchmarks, yet it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images. We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized and disentangled within VLM embeddings. Our analysis across multiple model families reveals a consistent vertical-distance entanglement: models conflate vertical image position with distance, mirroring the perspective bias of natural photographs. This bias produces a significant accuracy gap between perspective-consistent and counter-heuristic examples, and intensifies under data scaling even as overall benchmark accuracy improves. We further show that models with similar benchmark scores can exhibit different internal representations, and that these differences predict accuracy and robustness across diverse spatial reasoning benchmarks. To isolate this bias from evaluation-set skew, we introduce SpatialTunnel, a synthetic benchmark designed to expose spatial shortcut biases by removing common correlations present in natural images. Experiments confirm that the entanglement is model-intrinsic, and that models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial reasoning across diverse benchmarks. Code and benchmark are available on the project page: https://cheolhong0916.github.io/whyfarlooksup.github.io/.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at why our most advanced AI models sometimes struggle to "see" the world in three dimensions.
Sam: That's right. The paper investigates whether Vision-Language Models—systems that process both images and text—actually understand 3D space, or if they're just using clever shortcuts.
Alex: So, are they gaining a real sense of depth, or just pattern-matching their way through?
Sam: The researchers found that these models often confuse an object's vertical position in a 2D photograph with its actual 3D distance. Think about a photograph of a road. A car far away appears near the top of the image, while a car close to you appears near the bottom. Models trained on millions of photos pick up on that pattern—and start treating "higher in the frame" as a reliable signal for "farther away." The problem is, that shortcut breaks the moment you show them a scene that doesn't follow that rule.
Alex: That makes sense. It's like a student who memorized the answers to textbook diagrams but gets confused the moment you redraw the picture.
Sam: Exactly. The researchers call this "vertical-distance entanglement." It means the model has fused two separate ideas—where something sits in the frame, and how far away it actually is—into one inseparable rule. It's not reasoning about depth. It's just reading the height of an object in the photo and calling that distance.
Alex: So how did they actually prove this was a shortcut and not genuine understanding?
Sam: They developed a technique called "representation-level probing." Think of it like opening up the model's internal wiring diagram. Rather than just asking the model questions and checking its answers, they looked directly at the model's internal states—the patterns of numbers it uses to represent what it's seeing. They swapped object positions around and watched whether those internal states changed in a way that tracked real distance, or just tracked height in the frame.
Alex: Oh—so if the model truly understood depth, its internal representation should stay consistent with actual distance, even when the image looks unusual?
Sam: Precisely. And to make that test rigorous, they built something called "SpatialTunnel"—a controlled digital environment where they could place objects at any real-world distance while putting them anywhere in the frame. That way, height in the image and actual depth were completely independent. The models couldn't rely on the usual shortcut.
Alex: And what did they find when the shortcut was removed?
Sam: The bias ran deep. Even models that had been trained on more data, or built with different internal designs, still failed when SpatialTunnel placed objects in positions that violated their learned assumptions. Simply adding more training images doesn't fix the underlying problem, because the model has already organized its internal map of the world around the wrong rule.
Alex: So doesn't that make these models genuinely brittle? They perform well on standard photos, but fall apart the moment the scene is slightly unusual?
Sam: That's the concern, yes. The researchers found that the models which held up best were the ones whose internal structure kept distance and height as clearly separate concepts—like two distinct filing cabinets rather than one jumbled drawer. It's the difference between a student who has memorized a map and one who actually understands how navigation works. The first one is lost the moment a road is closed. The second one finds a different route.
Alex: So the accuracy numbers we see on standard benchmarks are, in a sense, misleading?
Sam: That's the key takeaway from the paper. Strong benchmark scores can mask a genuine lack of spatial understanding, because standard tests don't deliberately break the shortcuts. True robustness—the ability to handle unusual or counter-intuitive scenes—only appears when a model has built a genuinely independent internal representation of depth. The paper suggests that's the direction the field needs to move in, if we want AI systems that perceive the world reliably, not just in familiar conditions.
Alex: It's a useful reminder that a high score on a test doesn't always mean you understand the subject. Sometimes it just means you've learned the test.
Sam: And in AI, the gap between those two things can matter quite a lot—especially as these systems get used in situations where the world doesn't cooperate with their assumptions.
Alex: Thanks for listening to ResearchPod.