[论文] DreamX: Video Diffusion Model is a Universal Latent Action Learner

论文概要

研究领域: CV 作者: Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong et al. 发布时间: 2026-11-03 arXiv: 2611.05377

中文摘要

从视频中学习通用的动作表示，需要建模跨不同执行保持一致、但对形态和外观变化不变的动态特性。现有的隐式动作模型从视觉转移中推断动作，但其表示往往将底层动作与无关的视觉变化纠缠在一起，限制了跨形态迁移。我们提出 DreamX，将预训练的视频扩散模型重新用作通用的隐式动作学习器。该方法从视频模型丰富的时空特征中提取紧凑的隐式动作，将每个隐式视频帧压缩 300 倍（1200 → 4 个 token）。借助视频模型的生成先验，这些隐式动作能够捕捉与视觉外观解耦的动作相关动态。所得隐式空间可作为跨形态（人类与机器人）、跨环境和跨动作参数化的共享动作接口。在 LARY-Bench 上，DreamX 在隐式动作模型中取得 SOTA 性能，并支持任意视频之间的转换以及跨形态、跨环境的零样本动作迁移。此外，该共享接口还支持在紧凑隐式动作空间中进行策略学习，降低基于视频扩散策略的计算成本。

原文摘要

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by \(\boldsymbol{300\times}\) (\(\boldsymbol{1{,}200 \rightarrow 4}\) tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

---

*自动采集于 2026-11-05*

#论文 #arXiv #AI #小凯