Cheolhong Min, Jaeyun Jung, Daeun Lee, Hyeonseong Jeon, Yu Su, Jonathan Tremblay, Chan Hee Song, Jaesik Park
4 min
Abstract
Vision-language models (VLMs) achieve strong performance on spatial reasoning benchmarks, yet it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images. We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized and disentangled within VLM embeddings. Our analysis across multiple model families reveals a consistent vertical-distance entanglement: models conflate vertical image position with distance, mirroring the perspective bias of natural photographs. This bias produces a significant accuracy gap between perspective-consistent and counter-heuristic examples, and intensifies under data scaling even as overall benchmark accuracy improves. We further show that models with similar benchmark scores can exhibit different internal representations, and that these differences predict accuracy and robustness across diverse spatial reasoning benchmarks. To isolate this bias from evaluation-set skew, we introduce SpatialTunnel, a synthetic benchmark designed to expose spatial shortcut biases by removing common correlations present in natural images. Experiments confirm that the entanglement is model-intrinsic, and that models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial reasoning across diverse benchmarks. Code and benchmark are available on the project page: https://cheolhong0916.github.io/whyfarlooksup.github.io/.
Alex: So doesn't that make these models genuinely brittle? They perform well on standard photos, but fall apart the moment the scene is slightly unusual?
Sam: That's the concern, yes. The researchers found that the models which held up best were the ones whose internal structure kept distance and height as clearly separate concepts—like two distinct filing cabinets rather than one jumbled drawer. It's the difference between a student who has memorized a map and one who actually understands how navigation works. The first one is lost the moment a road is closed. The second one finds a different route.
Alex: So the accuracy numbers we see on standard benchmarks are, in a sense, misleading?
Sam: That's the key takeaway from the paper. Strong benchmark scores can mask a genuine lack of spatial understanding, because standard tests don't deliberately break the shortcuts. True robustness—the ability to handle unusual or counter-intuitive scenes—only appears when a model has built a genuinely independent internal representation of depth. The paper suggests that's the direction the field needs to move in, if we want AI systems that perceive the world reliably, not just in familiar conditions.
Alex: It's a useful reminder that a high score on a test doesn't always mean you understand the subject. Sometimes it just means you've learned the test.
Sam: And in AI, the gap between those two things can matter quite a lot—especially as these systems get used in situations where the world doesn't cooperate with their assumptions.
Alex: Thanks for listening to ResearchPod.