X-Sim: Robots Learn Manipulation from a Single Human Video with Zero Teleoperation Data

Forum topic · 二一 · 2026-05-13

Summary

X-Sim, presented as an oral at CoRL 2025, introduces a cross-embodiment learning framework that trains real robots from a single RGBD video of a human performing a task, requiring no robot teleoperation data. The pipeline reconstructs a 3D scene from the video, tracks object trajectories, and uses object motion as an RL reward to train a policy, which is then distilled into an image-conditioned diffusion policy and deployed to real robots via online domain adaptation. The key insight is treating object motion — rather than mapping human joints to robot joints — as a universal, transferable cross-embodiment signal, sidestepping morphology differences. Across 5 manipulation tasks in 2 environments, X-Sim improves over hand-crafted tracking baselines by 30% and matches behavior cloning baselines that require 10x more demonstration data. The work suggests teaching robots what outcome objects should achieve, rather than how the robot should move.

Teaching robots manipulation tasks typically requires dozens or even hundreds of human demonstrations. X-Sim, presented as an oral at CoRL 2025, offers a fundamentally different path: a single video of a human doing the task, with zero robot teleoperation data.

Pipeline

RGBD human video → 3D scene reconstruction → object trajectory tracking → RL policy trained with object motion as reward → distillation into an image-conditioned diffusion policy → online domain adaptation for deployment on real robots.

Key Innovation

The core idea is to use object motion as a cross-embodiment universal signal. Instead of trying to map human joints to robot joints — which becomes nearly impossible when morphologies differ significantly — X-Sim focuses on *how objects were moved*. This signal is universal and transferable between humans and robots.

Results

5 manipulation tasks across 2 environments
30% improvement over hand-crafted tracking baselines
Matches behavior cloning baselines that require 10x more data

A fresh perspective: perhaps instead of teaching robots *how to move*, we should teach them *what outcome objects should be moved into*.

Reference: [X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real / CoRL 2025 Oral]