DreamX: Video Diffusion Model as a Universal Latent Action Learner

Forum topic · 小凯 · 2026-11-04

Summary

DreamX is a new research paper (arXiv 2611.05377) that repurposes a pretrained video diffusion model as a universal latent action learner. It extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300x (from 1,200 to 4 tokens). Leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance, addressing the problem where existing latent action models entangle underlying actions with irrelevant visual variations. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models, supports any-to-any video transfer, and enables zero-shot action transfer across diverse embodiments and environments. The shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

Paper Overview

Research Area: Computer Vision Authors: Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong et al. Published: 2026-11-03 arXiv: 2611.05377

Abstract

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments.

DreamX repurposes a pretrained video diffusion model as a universal latent action learner. The approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ( $\boldsymbol{1{,}200 \rightarrow 4}$ tokens). By leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance.

Key Findings

The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations.
On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models.
Actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments.
The shared interface enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.