Despite the remarkable prowess of Vision-Language Models (VLMs) in general multimodal tasks, they remain fundamentally ``flat'' when reasoning about the physical world. We argue that this spatial bottleneck stems from a profound dimensional mismatch: while VLMs are trained to interpret 2D projections, true spatial reasoning demands the recovery of latent 3D geometry and temporal continuity. To conquer this high-dimensional complexity, we advocate a shift from monolithic learning to a ``divide and conquer'' paradigm. We present FactoSR, a factorized reinforcement learning framework that explicitly interpret the dimensions collapsed by visual projection. At its core, FactoSR decomposes the monolithic problem of world-consistent reasoning into three orthogonal, geometric sub-objectives: planar correspondence ($XY$), depth consistency ($Z$), and temporal reversibility ($T$). By optimizing these verifiable constraints within a unified policy learning mechanism, we effectively transform an ill-posed projection recovery problem into a series of tangible reasoning steps. Extensive evaluations on multi-view and video benchmarks demonstrate that this elegant decomposition yields substantial gains in 3D and 4D reasoning, achieving a 5.9% boost on VSI-Bench and 4.5% on All-Angles-Bench. Our findings suggest that reinforcing explicit, factorized 4D consistency is a critical step toward evolving VLMs into robust, world-aware reasoners.
Alex: Welcome to another episode of ResearchPod. Today we're looking at Unfold The World, a paper that attacks a foundational problem in AI: why do Vision-Language Models struggle to reason about 3D space?
Sam: It's a deeper issue than it first appears. VLMs are essentially flat — trained on 2D projections, they discard the depth and temporal continuity that define the physical world. The authors argue this isn't just a data gap. It's a dimensional mismatch baked into how these models are trained.
Alex: So it's not that the models are missing some 3D training examples. It's that the 2D projection process itself destroys information that can't be recovered after the fact?
Sam: Right. When you project a 3D scene onto a 2D image, depth becomes ambiguous by construction. A VLM navigating a room isn't doing geometry — it's pattern-matching on appearance heuristics. It might learn that objects look smaller when they're far away, but that's a statistical regularity, not spatial reasoning. The authors argue you can't fix this by scaling up 2D training data. You need to change what the model is being asked to verify.
Alex: Which is where their proposed framework, FactoSR, comes in. What's the core idea?
Sam: The insight is divide and conquer. Instead of asking a model to reason about 4D space — X, Y, Z, and time — as a single monolithic objective, FactoSR decomposes it into three orthogonal reward signals, each targeting a specific geometric axis. Think of it like teaching someone to draw a 3D cube. Rather than grading only the final sketch, you separately grade perspective, shading, and whether they can mentally rotate the object and still produce a consistent result. Each sub-task is individually verifiable, and together they constrain the model toward genuine spatial understanding.
Alex: Can you be more concrete about what each reward is actually checking?
Sam: Sure. The XY reward enforces reprojection consistency — if the model claims a point is at a certain location in one view, does it land where geometry predicts when you reproject it into another view? The Z reward handles depth ranking — given two objects, which one is closer? That's a weaker signal than metric depth, but it's verifiable without dense ground truth. And the T reward enforces temporal cycle consistency — if you track a feature forward through time and then backward, do you end up where you started? Each of these is a falsifiable geometric claim. The model can't fake its way through with appearance heuristics alone.
Alex: And the optimization framework for enforcing all this is Group Relative Policy Optimization?
Sam: Yes. GRPO is the RL backbone. The key design choice is that these three rewards are kept orthogonal — they're not collapsed into a single scalar loss. That matters because if you blend them, the model can trade off one geometric constraint against another and still look good on aggregate. Keeping them separate forces the model to satisfy all three axes simultaneously, which is what compels it to build something like an implicit world model internally.
Alex: There's an obvious tension here though. Pushing hard on geometric constraints during fine-tuning could degrade the model's general multimodal capabilities. Did the authors address that?
Sam: They did, and it's one of the more careful design choices in the paper. They use a two-stage curriculum. First, supervised fine-tuning to anchor the model — essentially grounding it in the new geometric vocabulary before any RL pressure is applied. Then the reinforcement learning stage, where the factorized rewards do the heavy lifting. The idea is that SFT stabilizes the base capabilities, so the RL phase can push on spatial reasoning without catastrophic forgetting of general language and vision understanding.
Alex: That's a reasonable mitigation, though it does raise a question about what the ablation over those two stages looks like — whether the SFT stage is load-bearing or mostly scaffolding.
Sam: That's exactly the right place to push. The paper's central claim rests on the factorized reward structure being the key driver of spatial improvement. The SFT stage is framed as enabling rather than causal. But if you removed the factorization and kept everything else — the two-stage curriculum, the GRPO optimizer — you'd want to see that the gains collapse. That ablation is the load-bearing test of the core thesis.
Alex: And the broader architectural claim is that you don't need a dedicated 3D geometry module bolted onto the side of the model. The geometry emerges from the training pressure itself.
Sam: That's the bet the paper is making. Rather than building an explicit depth estimation head or a separate spatial reasoning engine, FactoSR tries to make geometric consistency a first-class training objective. If it works, the model learns to represent 3D structure implicitly — not because it was given a 3D supervision signal directly, but because it was forced to make predictions that are only consistent if it has internalized something like depth and motion.
Alex: Which is an elegant framing, but also where a careful referee would want to probe generalization. Does the implicit world model transfer to scenes and tasks outside the training distribution, or is it a well-calibrated approximation that breaks under distribution shift?
Sam: That's the open question the paper doesn't fully close. The factorized rewards are defined over specific geometric relationships — reprojection, depth ranking, cycle consistency — and those are grounded in particular data regimes. Whether the spatial reasoning that emerges is genuinely general, or whether it's a set of well-trained heuristics that happen to satisfy these constraints in-distribution, is something that follow-up work will need to stress-test.
Alex: So the contribution is a training paradigm that reframes 3D spatial reasoning as a set of verifiable geometric constraints, enforced through factorized RL rewards, rather than a supervised signal over explicit 3D representations.
Sam: That's a fair summary. The key move is making the problem decomposable. Once you can break 4D reasoning into independently verifiable sub-objectives, you can apply RL pressure to each axis without the model being able to shortcut through appearance alone. Whether that pressure produces robust spatial intelligence or a sophisticated approximation of it — that's the empirical question this line of work is now positioned to answer.
Alex: A genuinely interesting direction. Thanks for walking through the mechanism. And thanks to our listeners for joining us on ResearchPod.