Paper Overview
- Field: Robotics
- Authors: Yen-Jen Wang, Jiaman Li, Sirui Chen
- Published: 2026-07-01
- arXiv: 2507.00001
- 3D reconstruction: The pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments.
- Trajectory synthesis: Navigation and object-interaction trajectories are synthesized using privileged scene information, and paired egocentric observations are rendered after the fact.
- Scale: 48,000 paired trajectories are produced with no human intervention.
- Policy: A VLK policy predicts short-horizon whole-body kinematic trajectories, which a whole-body tracker converts into actions on a physical humanoid robot.
- Real-world evaluation: The system was tested on a physical Unitree G1 performing navigation and single-object carrying tasks.
Summary
Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale.
The authors address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes:
Conclusion
Synthetic interactions in reconstructed scenes provide effective supervision for sim-to-real, perception-based humanoid loco-manipulation.
Original Abstract
Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts...
--- *Auto-collected on 2026-07-01*