Visual Geometry Transformer in the Wild: Distractor-Free 3D Reconstruction

Traditional end-to-end 3D reconstruction models often struggle in real-world environments because they rely on the assumption that scenes are static and consistent across all views. In reality, transient distractors—such as moving people, vehicles, or occlusions—disrupt geometric consistency, leading to noisy or incorrect point clouds. This paper addresses the challenge of performing robust 3D reconstruction in these "in-the-wild" scenarios using a feed-forward approach, avoiding the slow, iterative optimization typical of NeRF or 3D Gaussian Splatting methods.

The authors introduce the Visual Geometry Transformer in the Wild (VGTW). The core innovation is a Distractor-aware Training (DAT) strategy that uses Low-Rank Adaptation (LoRA) to fine-tune the model's attention mechanism. By training with an auxiliary mask prediction head on a new dataset (RobustNeRF-Mask) containing pixel-level distractor annotations, the model learns to distinguish between stable scene components and transient distractors. The training process incorporates two specific losses: a Distractor Suppression Loss, which penalizes the influence of distractor-contaminated features, and a Cross-View Consistency Loss, which reinforces the geometric alignment of stable features across different images.

VGTW successfully produces clean, distractor-free 3D point clouds in a single forward pass. By explicitly suppressing attention to regions identified as distractors, the model prevents "feature pollution," where dynamic elements are incorrectly integrated into the static scene geometry. Experiments demonstrate that VGTW achieves state-of-the-art performance in handling dynamic, uncontrolled environments and shows strong zero-shot generalization capabilities, outperforming existing feed-forward methods that lack explicit distractor-handling mechanisms.