PyRUA-DataFarm: LLM Agents as Cost-Efficient Teleoperators for Real-Robot Data

Forum topic · 小凯 · 2026-11-04

Summary

PyRUA-DataFarm replaces human teleoperation with a coding LLM agent that rebuilds robot tasks in simulation from a single calibration episode. The agent writes the scene, an expert controller, and a success judge; a simulator provides exact actions while a video model conditioned on real images renders photorealistic, action-synchronized camera views—"real-ready" data deployable directly on real robots. Independent judges certify the expert and each rendered episode. On five real-robot tasks, policies trained solely on farm data match teleoperation-trained policies (93 vs. 84 successes per 100 trials) and generalize far better under unseen tablecloths and colored lighting (87 vs. 46). Task adaptation is fully automatic at $21 in API costs plus a $5 render contract; subsequent episodes cost $0.80 each. One hour of three-view data averages $107—below China's $117 teleoperation quote and North America's $290 labor cost.

Overview

Field: Machine Learning Authors: Kewei Zhang, Juncheng Ma, Guodong Lv, Duomin Wang, Feixiang Ruan, Ruiyang Si et al. Published: 2026-11-03 arXiv: 2611.05320

Key Points

Teleoperation remains the default way to collect robot demonstrations, but every episode consumes robot and operator time, and each new task starts from scratch. The authors ask whether a coding agent can take over the teleoperator's job.
In DataFarm, the agent receives a data request plus one calibration episode of the robot in the target scene, then rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge.
The simulator supplies exact actions, while a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently.
The output is "real-ready" data: synchronized camera views labeled with the robot's actions, on which a policy can be trained directly and deployed on a real robot.
Independent judges certify the expert and every rendered episode before admission to the dataset.

Results

On five real-robot tasks, policies trained only on farm data match policies trained on an equal number of teleoperated episodes (93 vs. 84 successes in 100 trials).
Under an unseen tablecloth and colored lighting, farm-data-trained policies keep 87 successes while teleoperation-trained policies drop to 46.
Adapting a task is fully automatic: on average $21 in API usage per task (paid once), plus $5 for a render contract.
Each subsequent episode costs about $0.80 for simulation, rendering, and review—with no robot or operator required.
One hour of three-view data for a task costs $107 on average, slightly below a $117 teleoperation quote in China and well below $290 in labor alone in North America.

---

*Auto-collected on 2026-11-05*