[论文] Perceive, Predict, and Plan: a Unified World Action Model for Egocentr...

论文概要

研究领域: CV 作者: Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song 发布时间: 2026-10-15 arXiv: 2610.22363

中文摘要

我们引入 P³-WAM，这是一个用于自我中心灵巧操作的统一世界动作模型，通过三种互补模式学习 3D 感知、动作条件预测和目标导向规划：感知（Perceive）从视频中重建铰接式手部运动和相机轨迹；预测（Predict）在给定动作下生成视觉未来；规划（Plan）在给定初始图像、语言指令和可选视觉目标的条件下，联合生成动作轨迹及其预期视觉结果。我们的模型将紧凑的结构化 3D 动作表示集成到预训练视频扩散 Transformer 中，采用混合 Transformer 架构，包含共享基础权重、模态特定适配和双向视觉-动作注意力。实验表明，统一训练相比单模式训练改善了所有三项能力。我们的模型在 3D 手部重建和动作条件视频预测方面在四个数据集上优于专业基线，同时生成的目标导向操作视频具有更强的指令遵循性和动作进展性。项目页面：https://p3-wam.pages.dev/

原文摘要

We introduce P\(^{3}\)-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action a...

#论文 #arXiv #CV #小凯