Paper Overview
Field: Computer Vision (CV) Authors: Jihan Yang, Zifan Zhao, Xichen Pan Published: 2025-05-23 arXiv: 2505.17387
Abstract (English translation)
Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate system that links observations across video frames. Yet this signal is largely ignored in multimodal large language models (MLLMs) for video understanding—they process frames as isolated 2D snapshots rather than persistent scenes as humans perceive them.
This paper revisits pose as a lightweight supervision signal and proposes Cambrian-P, a video MLLM equipped with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves significant improvements of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, shows good generalization across eight additional spatial and general video QA benchmarks, and as a byproduct achieves state-of-the-art streaming pose estimation on ScanNet.
Surprisingly, training on pseudo-labeled poses from in-the-wild videos further improves performance on general video QA benchmarks, suggesting that the benefits of pose extend beyond spatial reasoning. Taken together, these results position camera pose as a foundational signal for video models that reason about the physical world.
Key Contributions
- Frames camera pose as a lightweight but foundational supervision signal for video MLLMs
- Introduces per-frame learnable camera tokens plus a pose regression head
- Achieves 4.5-6.5% gains on VSI-Bench-style spatial reasoning benchmarks
- Generalizes across 8 additional spatial and general video QA benchmarks
- State-of-the-art streaming pose estimation on ScanNet as a byproduct
- Pseudo-labeled poses on in-the-wild videos improve general video QA as well
- arXiv: https://arxiv.org/abs/2505.17387