Paper Overview
Field: Computer Vision / Robotics Authors: Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, Tomás Lozano-Pérez arXiv: 2609.30249 Project page: https://yuyaoliu.me/projects/rapid
Summary
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration.
The iterative agentic loop of code refinement requires several key ingredients:
1. A testable task specification 2. Action primitives for robot execution 3. An interactive environment for program execution and verification
RAPID infers all three from the demonstration automatically.
Method
To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motions themselves:
- Action primitives are expressed as trajectory optimization procedures that achieve object-level motion effects.
- These primitives are composed via relational constraints that capture scene-specific geometric relations at runtime.
- Eight challenging contact-rich non-prehensile manipulation tasks in simulation.
- General grasping tasks from the LIBERO-Pro benchmark.
- Real-world deployment on a Franka arm, evaluated on all eight non-prehensile tasks.
- arXiv: https://arxiv.org/abs/2609.30249
- Project page: https://yuyaoliu.me/projects/rapid
Evaluation
In all experiments, RAPID shows strong performance and generalizes across object poses, shapes, materials, and environments.