Siyu Chen, Zehan Wang, Jiayang Xu, Yihan Wu, Jialei Wang, Junming Chen, Ziang Zhang, Yutong Ying, Zhou Zhao
9 min
Metric 3D foundation models aim to reconstruct the physical dimensions of a scene from RGB images. While researchers often evaluate these models using standard depth maps or camera-intrinsics estimation benchmarks, these metrics do not directly translate into accurate physical scale estimation. In this work, the authors introduce Pumpire, a unified benchmark designed to assess the fundamental capability of models to estimate distances between arbitrary 3D points.
The authors curated the pumpire-6k dataset, which consists of 100 diverse real-world scenes captured in both indoor and outdoor environments. Unlike previous datasets that rely on sensor-based depth maps—which are often inaccurate near reflective surfaces, thin structures, or transparent objects—the Pumpire ground truth is derived from manual physical measurements using a tape measure. This dataset provides a rigorous foundation for evaluating 29 different baseline configurations, spanning both image-level and video-level models, in both estimation and completion (depth-prior) settings.
The study uncovers several key findings regarding modern 3D foundation models. First, the authors demonstrate that raw sensor-based depth is highly reliable in benign environments, but it fails on complex structures where learned 3D foundation models show superior robustness. Second, they identify distinct behavior between model families: completion models (which incorporate depth priors) achieve high peak accuracy but are prone to catastrophic failures in challenging scenes, whereas estimation models maintain more consistent, though lower, accuracy across varying conditions. Finally, the research highlights that multi-view context and depth priors are complementary: multi-view estimation improves cross-view consistency, while depth priors anchor the reconstruction to a reliable physical scale.
We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors. In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models' point-to-point distance estimation capability, Pumpire directly assesses point-to-point distances from the reconstructed geometry. To this end, we collect a large-scale and diverse dataset (pumpire-6k) comprising 100 real-world scenes, each annotated with physically measured point-pair distances and containing 64 frames, for a total of 6,400 frames. Building on this dataset, we establish a holistic evaluation protocol that covers both image- and video-level 3D foundation models and enables direct assessment of point-pair distance errors and cross-setting comparison. We conduct extensive experiments across 29 baseline configurations of representative 3D foundation models and provide a comprehensive analysis of the results. By offering this benchmark, we target the more fundamental ability to perceive and estimate physical scale in the reconstructed 3D space, which prior evaluation protocols have largely overlooked. The project page can be found at https://pumpire.github.io/
Alex: How do they get a reference they can trust? If sensor depth is unreliable, deriving the answer from that sensor would undermine the test.
Sam: They place two stickers in each scene and measure the distance between their centers with a tape measure. The dataset covers one hundred real-world scenes, indoors and outdoors. They capture changing viewpoints, then retain a sampled sequence from each scene.
Alex: Transparent surfaces are a useful example here. The sensor might report the wrong geometry, but the physical separation between the sticker centers still gives a reference.
Sam: That’s why they include transparency, reflection, slender structures, and points near sharp edges. The dataset contains sixty-four hundred frames in total. Sticker locations are propagated with a tracking model, then checked and refined manually frame by frame.
Alex: What does a model have to produce for the comparison? Some methods output depth, while others reconstruct points directly.
Sam: The protocol extracts depth from either output. It uses the appropriate camera parameters to place each annotated pixel in three-dimensional camera space. Then it calculates the straight-line distance between the two recovered points and compares that with the tape measurement.
Alex: That gives accuracy from each viewpoint. But the physical distance stays fixed as the camera moves, so disagreement across views is another failure.
Sam: They measure that too: cross-view consistency, meaning agreement between measurements from different viewpoints. They report distance errors and the fraction of predictions within specified tolerances. Results are averaged within each scene first, then equally across scenes.
Alex: Before the results, what kinds of models are being compared? Otherwise a winner might simply have access to more information.
Sam: The study evaluates twenty-nine baseline configurations across four settings. Models use either a single image or multiple video frames. Each can estimate geometry from color images alone, or perform geometry completion, meaning it additionally receives sensor-depth measurements.
Alex: Those inputs make the comparisons interpretable, but not interchangeable. What happens when the scene is easy for the sensor?
Sam: Raw RealSense depth is a strong baseline on the normal scenes. At a five-percent distance tolerance, it succeeds roughly four times in five. The best single-image model using images alone reaches only about one time in three.
Alex: That makes adding depth sound compelling. Does the same advantage survive on the surfaces where depth sensing fails?
Sam: The ordering reverses on challenging scenes: most evaluated single-image configurations have lower relative error than the raw sensor. The authors identify twenty-four challenging scenes using unusually large variation in sensor-derived distances across viewpoints. That is a sensor-reliability split, not an independently defined difficulty scale.
Alex: So there are two comparisons to keep separate: beating the sensor, and remaining reliable when the sensor deteriorates. How do the model families differ?
Sam: Completion models are more precise on most frames, but can produce catastrophic errors on challenging regions. Those outliers inflate average relative error. Models using images alone sacrifice peak accuracy, but show more bounded errors across the difficulty split.
Alex: Then a single leaderboard could hide the trade-off. A model that wins on typical scenes might not be the one I want around reflective objects.
Sam: The rankings do change. Metric3D version two leads single-image estimation on normal scenes, while Depth Pro leads on challenging scenes. Depth Anything three leads video estimation on both subsets. But those are benchmark rankings, not demonstrated robot-task success rates.
Alex: If noisy sensor inputs cause those outliers, could providing less depth sometimes help? More measurements are not necessarily more useful measurements.
Sam: The authors randomly thin the depth inputs and find that denser inputs do not consistently improve single-image completion. Several models perform best at intermediate densities. Sparse inputs remove constraints, but also exclude much of the sensor noise, so both sensitivities matter.
Alex: Is that also true when you add depth to more views, rather than adding more points within one view?
Sam: In the two models tested for partial depth conditioning across views, supplying depth to more views consistently reduces relative error. The improvement differs substantially between models. So integrating measurements across views is a separate capability from tolerating noise within a depth map.
Alex: And combining video context with depth seems to offer two benefits: agreement across views and a physical scale anchor. How broadly does that conclusion hold?
Sam: In the aligned comparison of leading models on normal scenes, video estimation improves consistency more than distance accuracy. Depth-conditioned completion improves both, and combining depth with video performs best. But each scene tests one fixed sticker pair, not arbitrary points throughout every reconstructed surface.
Alex: That leaves a useful benchmark, with a specific scope. Who should read the full document, and where should they begin?
Sam: Researchers evaluating geometry models or choosing them for scale-sensitive applications should read it. Start with Pumpire Protocols to understand the camera-space choices. Then read Comprehensive Insights, especially the depth–camera coupling analysis and the accuracy-versus-robustness comparison. Remember that neither downstream task performance nor exhaustive scene geometry is tested here.
Alex: And what should everyone else carry away, even if they never use this dataset?
Sam: Evaluate the physical quantity you need, not just the ingredients used to predict it. A depth leaderboard is not a distance guarantee.
Alex: Thanks for listening.