Yijun Yang, Shenghe Zheng, Wenbo Li, Jianhui Liu, Haoze Sun, Yanbing Zhang, Jiaxiu Jiang, Lin Song, Haoyang Huang, Nan Duan, Lei Zhu
6 min
Abstract
Despite the remarkable prowess of Vision-Language Models (VLMs) in general multimodal tasks, they remain fundamentally ``flat'' when reasoning about the physical world. We argue that this spatial bottleneck stems from a profound dimensional mismatch: while VLMs are trained to interpret 2D projections, true spatial reasoning demands the recovery of latent 3D geometry and temporal continuity. To conquer this high-dimensional complexity, we advocate a shift from monolithic learning to a ``divide and conquer'' paradigm. We present FactoSR, a factorized reinforcement learning framework that explicitly interpret the dimensions collapsed by visual projection. At its core, FactoSR decomposes the monolithic problem of world-consistent reasoning into three orthogonal, geometric sub-objectives: planar correspondence ($XY$), depth consistency ($Z$), and temporal reversibility ($T$). By optimizing these verifiable constraints within a unified policy learning mechanism, we effectively transform an ill-posed projection recovery problem into a series of tangible reasoning steps. Extensive evaluations on multi-view and video benchmarks demonstrate that this elegant decomposition yields substantial gains in 3D and 4D reasoning, achieving a 5.9% boost on VSI-Bench and 4.5% on All-Angles-Bench. Our findings suggest that reinforcing explicit, factorized 4D consistency is a critical step toward evolving VLMs into robust, world-aware reasoners.
Alex: There's an obvious tension here though. Pushing hard on geometric constraints during fine-tuning could degrade the model's general multimodal capabilities. Did the authors address that?
Sam: They did, and it's one of the more careful design choices in the paper. They use a two-stage curriculum. First, supervised fine-tuning to anchor the model — essentially grounding it in the new geometric vocabulary before any RL pressure is applied. Then the reinforcement learning stage, where the factorized rewards do the heavy lifting. The idea is that SFT stabilizes the base capabilities, so the RL phase can push on spatial reasoning without catastrophic forgetting of general language and vision understanding.
Alex: That's a reasonable mitigation, though it does raise a question about what the ablation over those two stages looks like — whether the SFT stage is load-bearing or mostly scaffolding.
Sam: That's exactly the right place to push. The paper's central claim rests on the factorized reward structure being the key driver of spatial improvement. The SFT stage is framed as enabling rather than causal. But if you removed the factorization and kept everything else — the two-stage curriculum, the GRPO optimizer — you'd want to see that the gains collapse. That ablation is the load-bearing test of the core thesis.
Alex: And the broader architectural claim is that you don't need a dedicated 3D geometry module bolted onto the side of the model. The geometry emerges from the training pressure itself.
Sam: That's the bet the paper is making. Rather than building an explicit depth estimation head or a separate spatial reasoning engine, FactoSR tries to make geometric consistency a first-class training objective. If it works, the model learns to represent 3D structure implicitly — not because it was given a 3D supervision signal directly, but because it was forced to make predictions that are only consistent if it has internalized something like depth and motion.
Alex: Which is an elegant framing, but also where a careful referee would want to probe generalization. Does the implicit world model transfer to scenes and tasks outside the training distribution, or is it a well-calibrated approximation that breaks under distribution shift?
Sam: That's the open question the paper doesn't fully close. The factorized rewards are defined over specific geometric relationships — reprojection, depth ranking, cycle consistency — and those are grounded in particular data regimes. Whether the spatial reasoning that emerges is genuinely general, or whether it's a set of well-trained heuristics that happen to satisfy these constraints in-distribution, is something that follow-up work will need to stress-test.
Alex: So the contribution is a training paradigm that reframes 3D spatial reasoning as a set of verifiable geometric constraints, enforced through factorized RL rewards, rather than a supervised signal over explicit 3D representations.
Sam: That's a fair summary. The key move is making the problem decomposable. Once you can break 4D reasoning into independently verifiable sub-objectives, you can apply RL pressure to each axis without the model being able to shortcut through appearance alone. Whether that pressure produces robust spatial intelligence or a sophisticated approximation of it — that's the empirical question this line of work is now positioned to answer.
Alex: A genuinely interesting direction. Thanks for walking through the mechanism. And thanks to our listeners for joining us on ResearchPod.