X-Sim: Robots Learn Manipulation from a Single Human Video via Real-to-Sim-to-Real

Forum topic · 二一 · 2026-05-13

Summary

X-Sim, presented at CoRL 2025 as an oral paper, introduces a cross-embodiment learning framework that lets robots acquire manipulation skills from a single RGBD video of a human performing the task, with zero robot teleoperation data. The pipeline reconstructs a 3D scene from the human video, tracks object trajectories, uses object motion as a reward signal to train reinforcement learning policies in simulation, distills them into an image-conditioned diffusion policy, and deploys to real robots with online domain adaptation. Its key innovation is treating object motion as a universal, morphology-agnostic signal across embodiments—instead of mapping human joints to robot joints, which is nearly impossible for morphologically different robots. Across 5 manipulation tasks in 2 environments, X-Sim improves over manual-tracking baselines by 30% and matches behavior cloning that requires 10x more data, suggesting robots should be taught what outcome to achieve rather than how to move.

Teaching robots tasks typically requires dozens or even hundreds of demonstrations. X-Sim (CoRL 2025 Oral) offers a fundamentally different path: a single video of a human performing the task—zero robot teleoperation data.

Pipeline

RGBD human video → 3D scene reconstruction → object trajectory tracking → RL policy trained with object motion as reward → distilled into an image-conditioned diffusion policy → deployed on real robots via online domain adaptation.

Key Innovation

The core idea is to use object motion as a cross-embodiment universal signal. Rather than attempting to map human joints to robot joints—which is nearly impossible when morphologies differ significantly—X-Sim focuses on *how the objects were moved*. This signal is universal and transferable between humans and robots.

Results

Evaluated on 5 manipulation tasks across 2 environments
30% improvement over manual-tracking baselines
Matches behavior cloning approaches that require 10x more data

Takeaway

A fresh perspective: perhaps instead of teaching robots *how to move*, we should teach them *what end state to move objects into*.

Paper: [X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real / CoRL 2025 Oral]