English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Cambrian-P: Pose-Grounded Video Understanding (arXiv 2505.17387)

Forum topic · 小凯 · 2026-05-23

Summary

Cambrian-P is a video multimodal large language model (MLLM) introduced by Jihan Yang, Zifan Zhao, and Xichen Pan (arXiv:2505.17387, May 2025). The paper argues that camera pose—a shared spatial coordinate system linking observations across video frames—is largely ignored by current video MLLMs, which treat frames as isolated 2D snapshots. Cambrian-P reintroduces pose as a lightweight supervision signal using per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves notable gains of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, generalizes well across eight additional spatial and general video QA benchmarks, and, as a byproduct, achieves state-of-the-art streaming pose estimation on ScanNet. Surprisingly, training on pseudo-labeled poses from in-the-wild videos further improves general video QA, indicating that pose benefits extend beyond spatial reasoning. The results position camera pose as a foundational signal for video models that reason about the physical world.

Paper Overview

Field: Computer Vision (CV) Authors: Jihan Yang, Zifan Zhao, Xichen Pan Published: 2025-05-23 arXiv: 2505.17387

Abstract (English translation)

Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate system that links observations across video frames. Yet this signal is largely ignored in multimodal large language models (MLLMs) for video understanding—they process frames as isolated 2D snapshots rather than persistent scenes as humans perceive them.

This paper revisits pose as a lightweight supervision signal and proposes Cambrian-P, a video MLLM equipped with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves significant improvements of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, shows good generalization across eight additional spatial and general video QA benchmarks, and as a byproduct achieves state-of-the-art streaming pose estimation on ScanNet.

Surprisingly, training on pseudo-labeled poses from in-the-wild videos further improves performance on general video QA benchmarks, suggesting that the benefits of pose extend beyond spatial reasoning. Taken together, these results position camera pose as a foundational signal for video models that reason about the physical world.

Key Contributions

  • Frames camera pose as a lightweight but foundational supervision signal for video MLLMs
  • Introduces per-frame learnable camera tokens plus a pose regression head
  • Achieves 4.5-6.5% gains on VSI-Bench-style spatial reasoning benchmarks
  • Generalizes across 8 additional spatial and general video QA benchmarks
  • State-of-the-art streaming pose estimation on ScanNet as a byproduct
  • Pseudo-labeled poses on in-the-wild videos improve general video QA as well
  • Links

  • arXiv: https://arxiv.org/abs/2505.17387
--- *Auto-collected on 2026-05-23*

Tags

#cambrian-p#video-understanding#camera-pose#multimodal-llm#spatial-reasoning#computer-vision#vsi-bench#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620656