Gen4P: Video Generators are 4D Perceivers

Forum topic · 小凯 · 2026-10-17

Summary

Gen4P is a unified framework for recovering 4D geometry and motion from monocular video, introduced by Yitong Xia, Zeren Jiang, and Chuanxia Zheng (arXiv:2610.22371). The key insight is that a pretrained video generator can be repurposed as a feed-forward 4D perceiver without iterative denoising. Gen4P processes clean video latents in a single encoder pass and predicts requested 4D attributes through a lightweight cross-attention decoder. An on-demand query formulation enables flexible spatial sampling, while proposed continuous-time queries add temporal flexibility. To strengthen geometric knowledge, the authors incorporate a pretrained video-depth LoRA as a frozen geometry prior via a plug-in solution, improving performance without adding trainable parameters. Gen4P achieves state-of-the-art or competitive results on 3D point tracking, video depth estimation, and camera pose estimation, and qualitative results on in-the-wild videos show it can track complex motion while preserving fine structure.

论文概要

研究领域: CV 作者: Yitong Xia, Zeren Jiang, Chuanxia Zheng 发布时间: 2026-10-15 arXiv: 2610.22371

Abstract

We introduce Gen4P, a unified framework for recovering 4D geometry and motion from monocular video. We show that a pretrained video generator can be repurposed into a feed-forward 4D perceiver without iterative denoising. Building on on-demand querying, Gen4P processes clean video latents in a single encoder pass and predicts the requested 4D attributes through a lightweight cross-attention decoder. The on-demand query formulation enables flexible spatial sampling, and our proposed continuous-time queries further provide temporal flexibility. To strengthen geometric knowledge, we propose a straightforward plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior, improving overall performance without adding extra trainable parameters. Gen4P achieves state-of-the-art or competitive performance on 3D point tracking, video depth, and camera pose estimation tasks. Qualitative results on in-the-wild videos demonstrate its ability to track complex motions while maintaining fine structures.

Key Points

Repurposing, not retraining from scratch: A pretrained video generator acts as a feed-forward 4D perceiver, eliminating the need for iterative denoising.
On-demand querying: Clean video latents are encoded once; a lightweight cross-attention decoder predicts the requested 4D attributes (e.g., point trajectories, depth, camera pose).
Spatial and temporal flexibility: On-demand queries allow flexible spatial sampling, and continuous-time queries provide temporal flexibility.
Frozen geometry prior: A pretrained video-depth LoRA is plugged in as a frozen prior, boosting performance with zero extra trainable parameters.
Results: State-of-the-art or competitive performance on 3D point tracking, video depth, and camera pose estimation; strong qualitative results on in-the-wild videos.