Cambrian-P: Pose-Grounded Video Understanding via Camera Tokens

Forum topic · 小凯 · 2026-05-23

Summary

This paper introduces Cambrian-P, a video multimodal large language model (MLLM) that incorporates camera pose as a lightweight supervision signal for video understanding. The authors argue that standard video MLLMs treat frames as isolated 2D snapshots, ignoring the spatial coordinate system defined by camera position and orientation. Cambrian-P augments a video MLLM with per-frame learnable camera tokens and a pose regression head, trained using a carefully designed sampling scheme. The model achieves 4.5-6.5% gains on VSI-Bench spatial reasoning benchmarks, generalizes well across eight other spatial and general video QA benchmarks, and produces state-of-the-art streaming pose estimation on ScanNet as a byproduct. Notably, training on pseudo-labeled poses from in-the-wild videos also improves general video QA performance, suggesting pose supervision benefits extend beyond spatial reasoning. Overall, the work positions camera pose as a foundational signal for video models reasoning about the physical world.

Cambrian-P: Pose-Grounded Video Understanding

Research Area: Computer Vision (CV) Authors: Jihan Yang, Zifan Zhao, Xichen Pan Published: 2025-05-23 arXiv: 2505.17387

Overview

Camera pose is important. The position and orientation of each viewpoint define a shared spatial coordinate system that links observations across video frames. However, this signal is largely ignored in multimodal large language models (MLLMs) used for video understanding, which process frames as isolated 2D snapshots rather than as the persistent scenes humans perceive.

This paper revisits pose as a lightweight supervision signal and proposes Cambrian-P, a video MLLM equipped with per-frame learnable camera tokens and a pose regression head. Through a carefully designed sampling scheme, the model achieves substantial improvements of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, and generalizes well across another eight spatial and general video QA benchmarks. As a byproduct, it also achieves state-of-the-art streaming pose estimation on ScanNet.

Key Findings

Pose as supervision: Camera pose provides a lightweight, effective training signal that most video MLLMs currently overlook.
Spatial reasoning gains: 4.5-6.5% absolute improvement on VSI-Bench and strong generalization across eight other video QA benchmarks.
Pose estimation bonus: Achieves state-of-the-art streaming pose estimation on ScanNet without dedicated optimization.
Wild video scaling: Training on pseudo-labeled poses from in-the-wild videos further boosts general video QA performance, indicating pose benefits extend beyond spatial reasoning.
Takeaway: Camera pose should be treated as a foundational signal for video models reasoning about the physical world.

Resources

Paper: https://arxiv.org/abs/2505.17387