Multi-View Video Diffusion Policy (MV-VDP): A 3D Spatio-Temporal-Aware Video Action Model

Forum topic · 小凯 · 2026-04-06

Summary

Researchers from the Institute of Automation, Chinese Academy of Sciences, Tsinghua University, and Xi'an Jiaotong University propose MV-VDP (Multi-View Video Diffusion Policy), a robot manipulation framework that unifies 3D spatial perception and temporal dynamics prediction in a video diffusion model. MV-VDP projects 3D point clouds onto multiple orthogonal views and jointly predicts future multi-view RGB videos and heatmap videos encoding end-effector trajectories. Built on the Wan2.2 video foundation model, it achieves strong data efficiency: on Meta-World with only 5 demonstrations per task it reaches 89.1% average success rate, far surpassing Diffusion Policy (37.7%) and DreamZero (61.1%). In real-world experiments with a Franka Research 3 arm and three ZED 2i depth cameras, it achieves 57.1% success with only 10 demonstrations per task, outperforming DP3, π0.5, UVA, and BridgeVLA. Ablations show video pre-training is essential (success drops to 4.6% without it), and predicted videos provide interpretability, reducing collisions from 6/140 to 0/140. Main limitations are inference speed (~4.6s for 24 frames on an A100) and 256×256 heatmap resolution.

Overview

Researchers from the Institute of Automation, Chinese Academy of Sciences, together with Tsinghua University and Xi'an Jiaotong University, propose MV-VDP (Multi-View Video Diffusion Policy), a 3D spatio-temporal-aware video action model for robot manipulation. Its core idea: let the robot simultaneously perceive 3D spatial structure and model temporal dynamics, similar to how humans use two eyes to observe the world and predict the future. Remarkably, it can learn complex tasks from as few as 10 demonstration trajectories.

Background

Traditional robot policies decide actions from single 2D frames, lacking depth perception and any notion of motion, much like catching a frisbee with closed eyes.
Diffusion models learn to progressively denoise corrupted data; video diffusion models additionally learn spatio-temporal regularities — how objects move and scenes evolve.

Method

Multi-view projection: 3D point clouds captured by depth cameras are projected onto multiple orthogonal views, implicitly encoding 3D structure priors.
Heatmap actions: end-effector positions are converted into heatmap images (Gaussian blobs marking positions), making them processable by video diffusion models alongside RGB.
Joint prediction: the model simultaneously predicts future multi-view RGB videos (how the environment will evolve) and heatmap videos (how the robot should move), aligning action learning with pre-trained video representations.
Built on the Wan2.2 video foundation model, leveraging internet-scale video pre-training.

Why it works

1. Representation alignment — heatmap videos let the policy reuse video pre-training knowledge about physics and causality. 2. Interpretability — predicted videos can be previewed before execution; manual checking of predicted videos reduced collisions from 6/140 to 0/140. 3. Data efficiency — pre-encoded world knowledge means only ~10 demonstrations per task are needed.

Results

Meta-World (5 demos per task):

| Method | Avg. success rate | |---|---| | BC-Scratch | 26.2% | | Diffusion Policy | 37.7% | | AVDC | 58.9% | | DreamZero | 61.1% | | Track2Act | 67.4% | | MV-VDP | 89.1% |

Real-world (Franka Research 3 + 3× ZED 2i, 10 demos per task): tasks include Put Lion, Push-T, Scoop Tortilla, plus generalization variants (new background, elevated objects, dimmed lighting, novel object category).

| Method | Avg. success rate | |---|---| | DP3 | 0% | | π0.5 | 1.4% | | UVA | 5.7% | | BridgeVLA | 41.4% | | MV-VDP | 57.1% |

Robustness: performance shifts less than 3.3% under large hyperparameter perturbations (RGB loss weight, heatmap std, diffusion steps 1–50).

Ablations

Full MV-VDP: 89.1%; without LoRA: 87.4%; channel-dim concatenation: 81.1%; heatmap-only (no RGB prediction): 61.1%; no video pre-training: 4.6%.
Conclusion: video prediction of environment dynamics and video foundation-model pre-training are both critical; view-dimension concatenation preserves information better than channel concatenation.

Limitations

Inference takes ~4.6s for a 24-frame action chunk on an NVIDIA A100; TurboDiffusion integration is planned for 100–200× speedup.
Heatmap resolution is 256×256 (~4mm per pixel).
Currently uses fixed camera views; adaptive view selection is future work.

References

1. Li P, Chen Y, Xu Y, et al. Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model. arXiv:2604.03181, 2026. 2. Black K, et al. π0.5: a vision-language-action model with open-world generalization. arXiv:2504.16054, 2025. 3. Chi C, et al. Diffusion policy: Visuomotor policy learning via action diffusion. IJRR, 2025. 4. Yu T, et al. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRL, 2020. 5. Wu H, et al. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv:2312.13139, 2023.