XSkill: Teaching Robots to Learn Skills from Human Videos Across Embodiments

Forum topic · 小凯 · 2026-03-14

Summary

XSkill, presented at CoRL 2023 by researchers from Columbia University and JP Morgan AI Research, is a framework that enables robot manipulators to discover and execute skills learned from unlabeled human videos, bridging the embodiment gap between human bodies and robotic arms. The framework has three components: skill discovery, which uses contrastive learning with temporal cycle-consistency to build a shared skill embedding space and extracts skill prototypes via vector quantization; skill transfer, which uses a conditional diffusion policy conditioned on skill prototypes to generate robot actions, handling multimodal action distributions; and skill composition, coordinated by a Skill Alignment Transformer (SAT) that aligns robot progress with human demonstrations by task progress rather than wall-clock time, supporting retries on failure. In real Franka Panda kitchen experiments with long-horizon multi-step tasks never seen in training, XSkill achieved 76.7%-81.7% zero-shot success rates, far exceeding baselines like end-to-end imitation learning and XIRL (often below 20%). The work represents a shift from copying trajectories to understanding skill intent, promising dramatically improved data efficiency by leveraging internet-scale unlabeled human video.

XSkill (Cross Embodiment Skill Discovery) is a framework from Columbia University and JP Morgan AI Research (CoRL 2023) that lets a robot learn skills from unlabeled human videos and execute them with its own, very different, body. This article is an in-depth Chinese-language explainer of the method; below is a structured English summary.

The Problem: The Embodiment Gap

Humans have 10 dexterous fingers (~26 DoF), rich touch/proprioception, and evolutionary motor priors; a typical robot arm has a 2-finger gripper with few DoF, camera-plus-encoder sensing, and rigid motor-driven control. This is the embodiment gap.
Imitation learning traditionally requires expensive teleoperated robot demonstrations; meanwhile, YouTube-scale human video is abundant but unlabeled and embodied differently.

Key points

Skill prototypes: XSkill's core insight is that humans and robots can perform the "same" skill at an abstract level (e.g., grasp-press-push for opening a door), even if the physical execution differs. Skills are represented as vectors in a shared embedding space — a "multi-embodiment dictionary."
Skill discovery: A temporal skill encoder maps short video clips to skill vectors. Self-supervised contrastive learning with temporal cycle-consistency pulls matching human/robot clip pairs together without any labels. Vector quantization maintains a codebook of skill prototypes (tens of prototypes worked well for kitchen tasks).
Skill transfer: A conditional diffusion policy generates robot action sequences conditioned on skill prototypes. Unlike deterministic policies, diffusion models capture multimodal action distributions (e.g., reaching left or right around an obstacle rather than averaging into failure).
Skill composition: Given a human demonstration of a novel long-horizon task, XSkill parses it into a sequence of reusable skill prototypes and executes them. A Skill Alignment Transformer (SAT) aligns by task progress rather than timestamps, so the robot switches skills only after completing the current one, and supports retries when a grasp fails.
Results: In simulation (MetaWorld) and a real Franka Panda kitchen setup, XSkill outperformed end-to-end imitation learning, XIRL, and video-prediction baselines. On real-world multi-step combination tasks never seen in training (e.g., open drawer → take out cloth → open microwave → place cloth → close), XSkill achieved 76.7%–81.7% zero-shot success, versus often below 20% for baselines.
Failure modes: perception errors in object localization, ambiguous skill boundaries confusing SAT alignment, and limits of the retry mechanism after repeated failures.

Significance

Shifts imitation learning from copying trajectories to understanding skill intent, enabling generalization to new task compositions.
Greatly improves data efficiency by exploiting unlabeled internet-scale human video instead of costly robot teleoperation data.
A step toward general-purpose robots that compose learned skill primitives for unseen tasks.

References

1. Xu, M., Xu, Z., Chi, C., Veloso, M., & Song, S. (2023). XSkill: Cross Embodiment Skill Discovery. *CoRL 2023*, PMLR. https://proceedings.mlr.press/v229/xu23a.html 2. Zakka, K., et al. (2021). XIRL: Cross-embodiment Inverse Reinforcement Learning. https://arxiv.org/abs/2106.09632 3. Chi, C., et al. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. https://arxiv.org/abs/2303.04137 4. Qin, Y., Su, H., & Wang, X. (2022). From One Hand to Multiple Hands. *IEEE RA-L*. https://ieeexplore.ieee.org/document/9811681 5. Radosavovic, I., et al. (2023). Robot Learning with Sensorimotor Pre-training. https://arxiv.org/abs/2306.10007