Pumpire: A Unified Benchmark for Metric Distance Estimation in 3D Foundation Models

Forum topic · 小凯 · 2026-10-11

Summary

Pumpire is a unified benchmark for evaluating the metric point-pair distance estimation capability of both image-level and video-level 3D foundation models, with or without depth priors. Unlike prior approaches that evaluate depth and camera intrinsics separately, or assess point clouds with geometric similarity metrics that fail to directly reflect point-to-point distance accuracy, Pumpire measures point-to-point distances directly from reconstructed geometry. The accompanying pumpire-6k dataset contains 100 diverse real-world scenes, each annotated with physically measured point-pair distances and 64 frames, totaling 6,400 frames. The benchmark establishes a holistic evaluation protocol covering image- and video-level 3D foundation models, enabling direct distance error assessment and cross-setting comparison. The authors benchmark 29 representative 3D foundation models and provide comprehensive analysis. The work targets the more fundamental capability of perceiving, estimating, and reconstructing physical scale in 3D space, which previous evaluation protocols largely overlooked. Paper: arXiv:2610.12423; project page: https://pumpire.github.io/.

Paper Overview

Field: Computer Vision Authors: Siyu Chen, Zehan Wang, Jiayang Xu, Yihan Wu, Jialei Wang, Junming Chen, Ziang Zhang, Yutong Ying, Zhou Zhao Published: 2026-10-08 arXiv: 2610.12423

Abstract

We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors. In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models' point-to-point distance estimation capability, Pumpire directly assesses point-to-point distances from the reconstructed geometry.

To this end, we collect a large-scale and diverse dataset (pumpire-6k) comprising 100 real-world scenes, each annotated with physically measured point-pair distances and containing 64 frames, for a total of 6,400 frames. Building on this dataset, we establish a holistic evaluation protocol covering both image-level and video-level 3D foundation models, supporting direct point-pair distance error evaluation and cross-setting comparison.

We conduct extensive experiments with baseline configurations of 29 representative 3D foundation models and provide comprehensive analysis. By offering this benchmark, we target the more fundamental capability of perceiving, estimating, and reconstructing physical scale in 3D space, which has been largely neglected by prior evaluation protocols.

Key Contributions

Unified evaluation: Assesses both image-level and video-level 3D foundation models under one protocol, including models with and without depth priors.
Direct metric: Evaluates point-to-point distances from reconstructed geometry, rather than indirect depth/intrinsics or geometric similarity metrics.
pumpire-6k dataset: 100 real-world scenes with physically measured point-pair ground-truth annotations, 64 frames per scene (6,400 frames total).
Benchmarks: Baseline evaluation of 29 representative 3D foundation models.

Project page: https://pumpire.github.io/

*Auto-collected on 2026-10-11.*