Paper Overview
- Field: Computer Vision (CV)
- Authors: Jihan Yang, Zifan Zhao, Xichen Pan
- Published: 2026-05-25
- arXiv: 2505.14485
- Problem: Video MLLMs typically lack explicit 3D spatial cues, treating frames as independent 2D images.
- Method: Add per-frame learnable camera tokens and a pose regression head to a video MLLM; train with a pose-aware sampling scheme.
- Spatial reasoning gains: +4.5–6.5% on VSI-Bench.
- Generalization: Improvements transfer to 8 other spatial and general video QA benchmarks.
- Pose estimation bonus: State-of-the-art streaming pose estimation on ScanNet.
- In-the-wild finding: Pseudo-labeled poses from unconstrained videos also boost general video QA, indicating pose supervision is broadly useful.
- Takeaway: Camera pose is positioned as a fundamental signal for video models reasoning about the physical world.
Summary
Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate frame that ties observations across video frames together. Yet this signal is largely missing from multimodal large language models (MLLMs) used for video understanding—these models treat frames as isolated 2D snapshots rather than the continuous scenes humans perceive.
The paper revisits pose as a lightweight supervision signal and introduces Cambrian-P, an augmented video MLLM equipped with per-frame learnable camera tokens and a pose regression head. Through a carefully designed sampling scheme, the model achieves notable 4.5–6.5% improvements on the VSI-Bench spatial reasoning benchmark, generalizes to eight additional spatial and general video QA benchmarks, and as a byproduct delivers state-of-the-art streaming pose estimation on ScanNet.
Surprisingly, training on pseudo-labeled poses from in-the-wild videos further improves general video QA benchmarks, suggesting that the role of pose extends beyond spatial reasoning.