RoboDream: Compositional World Models for Scalable Robot Data Synthesis

Forum topic · 小凯 · 2026-06-03

Summary

RoboDream is a generalizable, embodiment-centric world model for scalable robot data synthesis, presented in an arXiv paper (2506.00003) by Junjie Ye, Rong Xue, and Basile Van Hoorick. Robot learning requires large-scale, diverse demonstrations, but real-world teleoperation data collection is expensive and slow. Video diffusion models offer a scaling path, yet existing generative approaches are limited to superficial visual augmentation or suffer from embodiment hallucinations producing physically infeasible motions. RoboDream synthesizes photorealistic demonstrations with novel objects, in novel scenes, and from novel viewpoints by anchoring generation to rendered robot motion while conditioning on explicit scene and object priors, decoupling trajectory execution from environment synthesis. This enables two data-scaling capabilities: retrieve-and-regenerate, reusing existing trajectories in new contexts without new motion data; and prop-free teleoperation, where operators manipulate in mid-air while the model hallucinates the target objects and scene, eliminating reset time. Real-world experiments show generated data consistently improves downstream policy performance and significantly reduces the need for real-world data across manipulation tasks.

论文概要

研究领域: CV 作者: Junjie Ye, Rong Xue, Basile Van Hoorick 发布时间: 2026-06-03 arXiv: 2506.00003

Abstract

Scaling robot learning requires large-scale, diverse demonstrations, yet real-world data collection via teleoperation remains prohibitively expensive and time-consuming. While video diffusion models offer a promising avenue for data scaling, existing generative approaches are often limited to superficial visual augmentation, or suffer from embodiment hallucinations that yield physically infeasible motions. We present a generalizable embodiment-centric world model that achieves scalable data generation by synthesizing photorealistic demonstrations with novel objects, in novel scenes, and from novel viewpoints. Our approach anchors generation to rendered robot motion while conditioning on explicit scene and object priors, effectively decoupling trajectory execution from environment synthesis. This formulation unlocks two powerful data-scaling capabilities:

1. Retrieve-and-regenerate — repurposing existing trajectories into entirely new contexts without new motion data. 2. Prop-free teleoperation — the operator manipulates in mid-air while the model hallucinates the target objects and scene, eliminating reset time.

Through real-world experiments, we demonstrate that the generated data consistently improves downstream policy performance and significantly reduces real-world data requirements across a variety of manipulation tasks.

Key points

An embodiment-centric world model that generates photorealistic robot demonstrations with novel objects, scenes, and viewpoints.
Generation is anchored to rendered robot motion and conditioned on explicit scene/object priors, avoiding embodiment hallucinations and physically infeasible motion.
Decouples trajectory execution from environment synthesis, enabling scalable data generation.
Two capabilities: retrieve-and-regenerate (reuse trajectories in new contexts) and prop-free teleoperation (operate in mid-air; model hallucinates objects and scene).
Real-world experiments show improved downstream policy performance and reduced real-world data needs across manipulation tasks.

---

*Auto-collected on 2026-06-03. Source: arXiv:2506.00003*