Scal3R: Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Forum topic · 小凯 · 2026-09-05

Summary

Scal3R is a new approach for online 3D reconstruction that addresses the failure of existing models on long videos. Current methods regress poses relative to a fixed first-frame anchor, forcing extrapolation beyond the training distribution, where accumulated drift leads to severe geometric collapse. The authors observe that per-frame depth remains stable during such failures—the backbone's local geometry stays intact while only the global pose head breaks down. Building on this decoupling, Scal3R reformulates online reconstruction as multi-reference relative pose querying: lightweight learnable tokens (~1% of parameters) are injected into a fully frozen backbone via asymmetric attention, querying poses against multiple past keyframes, combined with an online pose-graph optimization system with loop closure to suppress long-range drift. Scal3R converges in 8 hours on a single GPU, reduces average ATE by over 60% versus online baselines on KITTI, and achieves state-of-the-art performance on Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes. Paper: arXiv 2609.04201.

Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Research area: Computer Vision (CV) Authors: Chin-Yang Lin, Yang-Che Sun, Cheng Sun Published: 2026-09-03 arXiv: 2609.04201

Paper Summary

Online 3D reconstruction models perform poorly on long videos because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution. Small drifts continuously accumulate and amplify into severe geometric collapse.

However, the authors observe that during this failure process, per-frame depth remains stable—the backbone's local geometry stays intact, and only the global pose head collapses.

Method

Based on this decoupling observation, the authors propose Scal3R, which reformulates online reconstruction as multi-reference relative pose querying:

Lightweight learnable tokens (~1% of parameters) are injected into a fully frozen backbone via asymmetric attention.
The model queries poses relative to multiple past keyframes instead of a single fixed anchor.
An online pose-graph optimization system with loop closure suppresses long-range drift.

Results

Converges in 8 hours on a single GPU.
Reduces average ATE by over 60% compared to online baselines on KITTI.
Achieves state-of-the-art performance on Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes.

Original Abstract (excerpt)

> Online 3D reconstruction models perform poorly on long videos because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution. We introduce Scal3R, which reformulates online reconstruction as multi-reference relative pose querying using lightweight learnable tokens (~1% of parameters) injected into a frozen backbone via asymmetric attention.

---

*Auto-collected on 2026-09-05. Source: arXiv:2609.04201*