MV-VDP: Multi-View Video Diffusion Policy for 3D Spatio-Temporal-Aware Robotic Manipulation

Forum topic · 小凯 · 2026-04-06

Summary

MV-VDP is a multi-view video diffusion policy for robotic manipulation that jointly models the 3D spatial structure of the environment and its temporal evolution. Unlike most existing policies that rely on 2D visual observations and backbones pretrained on static image-text pairs—leading to high data requirements and poor understanding of environment dynamics—MV-VDP simultaneously predicts multi-view heatmap videos and RGB videos. This design aligns the representation format of video pretraining with action finetuning, and specifies not only what actions the robot should take but also how the environment is expected to evolve in response to those actions. Experiments demonstrate that MV-VDP can successfully execute complex real-world manipulation tasks using only ten demonstration trajectories. The paper by Peiyan Li, Yixiang Chen, Yuan Xu and colleagues is available on arXiv (2604.03181).

Paper Overview

Field: Computer Vision / Robotics Authors: Peiyan Li, Yixiang Chen, Yuan Xu, et al. arXiv: 2604.03181

Abstract

Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies overlook one or both. They typically rely on 2D visual observations and backbones pretrained on static image-text pairs, resulting in high data requirements and limited understanding of environment dynamics.

To address this, the authors introduce MV-VDP, a multi-view video diffusion policy that jointly models the 3D spatio-temporal state of the environment. The core idea is to simultaneously predict multi-view heatmap videos and RGB videos, which:

1. Align the representation format of video pretraining with action finetuning 2. Specify not only what actions the robot should take, but also how the environment is expected to evolve in response to those actions

Key Findings

Extensive experiments show that MV-VDP can successfully execute complex real-world tasks with only 10 demonstration trajectories, highlighting strong sample efficiency.
Joint multi-view heatmap + RGB video prediction enables explicit modeling of both 3D structure and temporal dynamics.

---

*Auto-collected on 2026-04-06.*