Dex-One2Many: Learning Dexterous Manipulation from a Single Human Video via Real-to-Sim-to-Real RL

Paper Overview

• Field: Computer Vision / Robotics
• arXiv: 2610.12470
• Authors: Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang, Seungjae Lee

Summary

Learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations. However, many recent methods predominantly imitate demonstrated motions, and such strict motion matching limits generalization to initial object poses, goal poses, and grasps not shown in the video. Conversely, discovering a policy via reinforcement learning (RL) allows broad generalization, but without prior guidance it struggles with high-dimensional exploration in complex, multi-stage tasks.

To address these coupled generalization and exploration challenges, the authors present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video.

Key Idea

Abstract the video into sequential scene graphs to guide RL:

• Scene graphs serve as generative constraints for sampling diverse reset states.
• They provide dense rewards for each stage of the task.
• Since scene graphs constrain inter-object relations rather than exact poses, reset states can cover object poses and grasps beyond those in the video.
• Dense rewards make exploration short and well-guided.

Results

• The model is trained entirely in simulation and transfers zero-shot to a real multi-finger robot hand.
• Across five tool-use and manipulation tasks, Dex-One2Many outperforms baselines by 6.5% on seen configurations.
• On unseen scenarios, the advantage expands to 71%.