[论文] One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for ...
研究领域: CV 作者: Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou 发布时间: 2026-11-19 arXiv: 2611.26617
论文概要
研究领域: CV 作者: Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou 发布时间: 2026-11-19 arXiv: 2611.26617
中文摘要
视频展示了世界如何变化,却很少揭示导致变化的动作。潜在动作学习从视觉转换中恢复这种监督信号,但往往需要从零开始学习底层动力学。我们提出:预训练视频生成器能否同时为逆动力学和正动力学提供先验?我们引入O²LaF,基于一个简单的原则:一个先验,两个动力学。我们从同一预训练视频DiT初始化两个模型并适配到各自角色:逆动力学用带可学习动作查询的前馈转换编码器,正动力学用因果的、以动作为条件的视频生成器。通过控制信息访问并通过紧凑瓶颈端到端优化两个动力学,O²LaF从无标签视频中学习可迁移的动作。在导航和机器人操作中,O²LaF在少样本动作探测上创造新SOTA,并实现更强的跨视觉上下文和跨形态零样本动作迁移。除了生成式视频先验,我们还将框架扩展到V-JEPA2等预测性表征先验,取得有竞争力的性能。我们的发现指向潜在动作学习的一个更广泛的范式:重新利用预训练视频模型来发现可迁移动作并预测其后果。
原文摘要
Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O2LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O2LaF learns transferable ac...
*自动采集于 2026-11-21*
#论文 #arXiv #CV #小凯