P3-WAM: A Unified World Action Model for Egocentric Dexterous Manipulation

Forum topic · 小凯 · 2026-10-17

Summary

P3-WAM is a unified world action model for egocentric dexterous manipulation that learns three complementary capabilities—3D perception, action-conditioned prediction, and goal-directed planning—in a single framework. The Perceive mode reconstructs articulated hand motion and camera trajectories from video; the Predict mode generates visual futures under prescribed actions; and the Plan mode jointly produces action trajectories and their anticipated visual consequences, conditioned on an initial image, a language instruction, and an optional visual goal. The model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer using a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Experiments show unified joint training improves all three abilities compared to single-mode training, outperforming specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction following and action progression. Project page: https://p3-wam.pages.dev/ (arXiv: 2610.22363).

Paper Overview

Research area: Computer Vision Authors: Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song Published: 2026-10-15 arXiv: 2610.22363

Introduction

We introduce P³-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes:

Perceive — reconstructs articulated hand motion and camera trajectories from video.
Predict — generates visual futures under prescribed actions.
Plan — jointly generates action trajectories and their anticipated visual consequences, conditioned on an initial image, a language instruction, and an optional visual goal.

Method

The model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture featuring:

Shared base weights
Modality-specific adaptation
Bidirectional visual-action attention

Results

Unified joint training improves all three capabilities compared to single-mode training.
The model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets.
It generates goal-directed manipulation videos with stronger instruction following and better action progression.