PRISM: Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

Forum topic · 小凯 · 2026-10-01

Summary

PRISM is a real-to-sim-to-real framework for teaching humanoid robots loco-manipulation skills like carrying diverse objects through visual imitation. The key challenge it addresses is that collecting diverse, high-quality interaction videos—clips showing a person's full body and unoccluded object interactions—is a practical barrier to scaling imitation learning. PRISM amplifies a handful of real exemplar videos into hundreds of diverse 'counterfactual' human-object interaction videos using video-to-video (V2V) generation. A contact-anchored real-to-sim pipeline then reconstructs human and object motions, retargeting imperfect video data into physically plausible trajectories. The intra-class variation across counterfactual videos enables training a single policy that generalizes to unseen objects within each category. Deployed on a real humanoid robot without any real-world fine-tuning, and using only onboard depth observations, the robot picks up, carries, and puts down objects including boxes, buckets, bins, and balls across novel instances, sizes, and initial configurations. The paper, by researchers including Zihan Wang, Pieter Abbeel, Rocky Duan, Jitendra Malik, and Angjoo Kanazawa, is available on arXiv (2609.38172).

Overview

Field: Computer Vision / Robotics Authors: Zihan Wang, Zhen Wu, Pieter Abbeel, Rocky Duan, Jitendra Malik, Carmelo Sferrazza, C. Karen Liu, Guanya Shi, Angjoo Kanazawa arXiv: 2609.38172

Abstract (translated from Chinese summary)

Teaching humanoid robots loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos—such as clips that clearly show a person's full body and unoccluded interactions with objects—poses a practical barrier to scaling this approach.

The authors propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set:

1. Counterfactual video generation: PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. 2. Contact-anchored real-to-sim pipeline: The pipeline reconstructs both human and object motions, retargeting this imperfect video data into physically plausible trajectories. 3. Generalizable policy: The intra-class variation across counterfactual videos enables training a single policy that generalizes to unseen objects within each object category.

Real-World Results

The policy was deployed on a real humanoid robot without any real-world fine-tuning. Using only onboard depth observations, the humanoid picks up, carries, and puts down objects such as boxes, buckets, storage bins, and balls—covering novel instances, varying sizes, and different initial configurations.

--- *Auto-collected on 2026-10-01*