[论文] Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipul...

论文概要

研究领域: CV 作者: Zihan Wang, Zhen Wu, Pieter Abbeel, Rocky Duan, Jitendra Malik, Carmelo Sferrazza, C. Karen Liu, Guanya Shi, Angjoo Kanazawa 发布时间: 2026-09-29 arXiv: 2609.38172

中文摘要

通过视觉模仿教学人形机器人搬运多样化物体等运动-操作（loco-manipulation）技能，是实现通用机器人的一条有前景的路径。然而，收集多样化、高质量的交互视频（如清晰展示人体全身及与物体无遮挡交互的片段）对这一方法的可扩展性构成了实际障碍。我们提出PRISM，一个真实到仿真再到真实（real-to-sim-to-real）的框架，通过将少量真实视频放大为大规模多样化训练集来克服这一限制。PRISM首先通过视频到视频（V2V）生成，从少量真实示例视频生成数百个多样化的'反事实'人-物交互视频。然后，我们的接触锚定真实到仿真流水线重建人体和物体运动，将这些不完美的视频数据重定向为物理上合理的轨迹。这些反事实视频中的类内变异使我们能够训练出在每类物品内对未见物体具有泛化能力的单一策略。我们在真实机器人上部署了该策略，无需任何真实世界微调。仅使用机载深度观测，我们的人形机器人就能拾取、搬运和放下箱子、桶、储物箱和球等物体，涵盖新颖实例、不同尺寸和初始构型。

原文摘要

Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach. We propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set. PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. Our contact-anchored real-to-sim pipeline then reconstructs both human and object motions, retargeting this imperfect video data ...

#论文 #arXiv #CV #小凯