Seokhyun Youn, Dahyeon Kye, Sung-Ho Bae, Jihyong Oh
5 min
Recent Vision Foundation Models (VFMs) provide impressive zero-shot 3D reconstruction capabilities but often lack explicit multi-view geometric consistency because such constraints are computationally expensive to enforce during pretraining. This paper asks whether a lightweight, plug-and-play test-time adaptation (TTA) pipeline can enforce these geometric constraints on frozen VFMs to improve their pose and depth estimation accuracy without requiring ground-truth labels.
The authors propose Self-Geometry, a TTA pipeline that uses 2D pixel correspondences—extracted at test-time by an external feature matcher—as pseudo ground-truth. The pipeline consists of three main components:
Self-Geometry consistently improves both camera pose and geometry estimation across six different VFMs (including VGGT, π³, and Depth Anything 3 variants) and four diverse benchmarks (7Scenes, ETH3D, ScanNet++, and HiRoom). The method is efficient, completing per-scene adaptation in under two minutes on a single GPU, and effectively resolves the pose-depth ambiguity inherent in standard reprojection-based supervision.
This work demonstrates that explicit geometric constraints can be effectively injected into frozen foundation models at test-time. By eliminating the need for ground-truth data or teacher-model distillation, Self-Geometry provides a flexible, model-agnostic way to enhance the geometric accuracy of 3D vision models for arbitrary, unseen scenes.
Recent Vision Foundation Models (VFMs) predict depth, camera pose, and pointmap in a single forward pass without per-scene optimization, achieving strong generalization. However, enforcing explicit multi-view geometric consistency, e.g., through bundle adjustment, is computationally costly and is thus not imposed during VFM pretraining, so such inconsistency can arise. To address this, implicit self-consistency derived from model outputs (e.g., pointmaps, features), though enforced at test-time in prior work, delivers inherently limited performance gain, especially on scenes where the pretrained VFM is highly inaccurate. In contrast to this implicit signal, we propose Self-Geometry, a plug-and-play test-time adaptation pipeline that directly imposes explicit multi-view geometric constraints using 2D pixel correspondences as pseudo ground-truth. Our proposed Self-Geometry consists of Geometric Disentanglement Optimization, which combines Multi-View Consistency and Epipolar Consistency losses with Gradient Disentanglement to prevent gradient conflict; Frame Angular-Neighbor, a view sampler based on SO(3) geodesic distances for lightly imposing these constraints; and Lightweight TTA, which adapts VFMs via LoRA. Our method achieves consistent improvements in both pose and geometry estimation across six VFMs (VGGT, $π^3$, DA3-Giant/Large/Base/Small) and four benchmarks (7Scenes, ETH3D, ScanNet++, HiRoom).
Alex: And this all happens while the camera is moving through a space?
Sam: Yes, and it needs to be fast. Rather than updating the entire AI model—which would be like rebuilding a radio from scratch every time you want a clearer signal—the system only adjusts a tiny fraction of the model's settings. Like turning a few fine-tuning knobs instead of rewiring the whole device. The entire process takes under two minutes per scene.
Alex: But before any of that learning happens, how does the system know which matched points are actually trustworthy?
Sam: That's a critical step. An external program finds matching points between photos, but it often gets things wrong—matching a shadow to a dark corner, for instance. So before the AI learns anything, the system runs a two-stage filter to weed out bad matches.
Alex: What does each stage do?
Sam: The first stage checks basic geometric possibility. Any point seen by two cameras must fall on a specific, predictable line of sight between them. If a matched point doesn't land on that line, it's geometrically impossible—so it gets discarded immediately. The second stage checks consistency across multiple views. If the AI's predicted distance and angle for a point don't hold up when you look at that same point from a third or fourth position, the math doesn't close the loop, and that point gets thrown out too. Only the matches that survive both filters are used to train the model.
Alex: So you're cleaning the data first, then managing how the AI learns from it, then keeping the whole thing lightweight enough to run in real time.
Sam: That's the design. And one more piece ties it together: the system is deliberate about which video frames it selects for training. It picks frames from meaningfully different angles—not just consecutive frames that look nearly identical. That way, the AI gets a well-rounded view of the space rather than a narrow slice of it.
Alex: So by combining reliable data, careful learning, and efficient computation, the system turns a pattern-matching model into something that actually respects the rules of physical space.
Sam: That's the core of it. The practical implication is meaningful—3D reconstruction that works on the move, without needing expensive pre-labeled data or a full retraining run. It's a significant step toward AI vision that holds up in the real world.
Alex: Thanks for walking us through it, Sam. And thanks to everyone listening to ResearchPod.