RLE-Bench: Harvard and Georgia Tech Release a 48-Task Robotics Engineer Exam for AI Coding Agents

Forum topic · 小凯 · 2026-09-21

Summary

Harvard SEAS and Georgia Tech researchers released RLE-Bench (Robot Learning Engineer Benchmark), an open-source benchmark testing whether AI coding agents can perform the full range of engineering tasks required to build and debug robots — not just write code. The benchmark contains 48 tasks across four domains: interactive control, policy development, perception and estimation, and mechanical design. Unlike existing robot evaluations that focus on control policies or end-to-end task completion, RLE-Bench tests the engineering layer in between. A flagship example asks an agent to design a mobile base supporting multiple robotic arms: code that passes simulation checks fails stability testing once real payloads are attached, exposing the gap between computational plausibility and physical validity. The benchmark enforces compute and time budgets, evaluates agents against hidden physical conditions (wind, load, sensor noise), and uses a continuous 0-100 score instead of pass/fail. The team, led by Harvard's Na Li and Georgia Tech's Bo Dai, plans a 2.0 version covering hardware design, system integration, debugging, high-fidelity simulation, safety, and real-hardware deployment. RLE-Bench positions itself as the first major open benchmark for the "robotics engineer" layer of embodied AI evaluation.

> *48 tasks / four exam domains / running in simulation ≠ standing in reality / RLE-Bench, released September 18, pushes AI from "writing code" to "building machines"*

Overview

A hummingbird flaps its wings 50 times per second to hover. To stabilize it, simply flapping faster won't work — it changes how it moves. Over the past year, AI coding agents have been like a hummingbird trying to hover by flapping faster: bigger models, longer contexts, more tokens, with SWE-bench Verified Pass@1 climbing from ~30% to ~80%. But making written code drive a real robot remains nearly the same problem as it was a year ago.

Harvard SEAS and Georgia Tech researchers released RLE-Bench (Robot Learning Engineer Benchmark) on September 18, an open-source benchmark asking whether AI coding agents can complete the full set of engineering tasks needed to build and debug robots.

The "robot learning engineer" role combines machine learning, robotics, control theory, software engineering, and hardware engineering in a single job. RLE-Bench opens version 1.0 of this role's qualification exam: 48 tasks, four domains.

Motivation: from "control policies" to "engineering tasks"

Current robot evaluation systems focus on two things: control policies (state → action) and high-level embodied tasks (natural-language instruction → completion). Neither tests "building the robot itself." As Li put it: existing benchmarks focus on learning and evaluating control policies; RLE-Bench takes a broader view — can an AI system complete the full set of engineering tasks needed to make robots actually work?

The four domains:

Interactive control: forward/inverse kinematics, trajectory planning, impedance control, force feedback loops. Strength: translating control laws into code. Weakness: simulation success ≠ real-hardware success.
Policy development: reinforcement learning, imitation learning, diffusion policies, world-model-driven policies. Strength: writing PyTorch/JAX policy code. Weakness: generalization in long-horizon tasks.
Perception & estimation: image denoising, point-cloud registration, IMU integration, SLAM loop closure, multi-sensor time synchronization. Strength: assembling pipelines from existing libraries. Weakness: silent failure when noise distributions mismatch training assumptions.
Mechanical design: link length selection, center-of-mass calculation for mobile bases, joint torque checks, mechanical interface matching. AI coding agents have almost no prior experience here.

Interactive control and policy development make up roughly half the tasks; perception/estimation and mechanical design roughly a quarter each (per the paper and public materials; exact task counts should be confirmed against the published paper).

Key example: reachable on paper, tips over under load

In the benchmark's flagship task, an agent must design a general-purpose mobile base supporting multiple arms to reach objects on shelves at different heights. The generated code lets the arms "reach" all target positions — correct statically and in simulation. But once the arms carry real loads (e.g., a 5 kg tool), the base fails stability testing and the robot tips over.

Co-author Haitong Ma (Harvard SEAS graduate student) explained: this is exactly where physical reasoning becomes critical — the agent's output is computationally plausible, but important failure modes emerge once the full system's physics are considered.

The point isn't that "AI can't build robots"; it's that AI-generated designs can be right at one layer and wrong at another, and the wrong layer only surfaces when physical constraints are explicitly applied. Code-level pass ≠ physics-level pass.

How the benchmark runs

RLE-Bench constrains agents in three ways:

1. Compute and time budgets — agents may iterate (read errors, fix code, re-run) within a fixed budget, distinguishing "solves it in reasonable time" from "solves it eventually." 2. Hidden physical conditions — wind loads, temperature, load fluctuations, and sensor noise distributions are hidden from the agent and evaluated independently afterward, separating targeted optimization from genuine robustness. 3. Continuous 0-100 scoring — fine-grained scores distinguish near-misses from real gaps, closer to real engineering assessment than binary pass/fail.

Team

Na Li — Winokur Family Professor of Electrical Engineering and Applied Mathematics, Harvard SEAS; feedback control, multi-agent systems, RL theory.
Bo Dai — Assistant Professor, Georgia Tech; machine learning theory, reinforcement learning, Bayesian methods.
Students: Haitong Ma (Harvard), Chenxiao Gao (Georgia Tech), Rushi Qiang (Georgia Tech). Ma is the primary spokesperson for the benchmark's design philosophy.

Positioning among benchmarks

Robot benchmarks over the past year fall roughly into three layers: control policy evaluation (ML view), end-to-end task completion (embodied foundation model view), and engineering-task evaluation (robotics engineer view). RLE-Bench is currently the most prominent open-source benchmark in the third category; several embodied-AI companies reportedly use similar benchmarks internally, but RLE-Bench is the only public one (per author's cross-check of public materials, September 2026).

What's not yet disclosed

The contents of the other 47 tasks (only the mobile-base example is public).
Baseline results for specific LLMs — no public numbers yet for which models perform how.
Sim-to-real transfer: version 1.0 runs only in simulation. Li called RLE-Bench 1.0 "just a starting point."
Direct score comparisons with embodied benchmarks like Genmanip or BridgeData (different task counts, scoring, and costs).

Roadmap to 2.0

Li outlined six expansion directions: hardware design (mechanical structures, electrical interfaces, thermal design), system integration, debugging across layers, high-fidelity simulation setup, safety under fault modes, and deployment to real hardware. The intent is to grow RLE-Bench from an exam into a shared coordinate system for the robotics engineering community.

Three takeaways

1. The benchmark's value is the coordinate system, not the score. Whether agents score 30 or 50 matters less than giving industry a shared language for "how far is AI from building working robots" — the role SWE-bench Verified played for software engineering. 2. 48 simulation tasks are the minimum viable version of engineering-task benchmarks, opening questions like "why does my model score 80 on public VLA leaderboards but 30 on RLE-Bench?" 3. Physical reasoning is the key gap for the next stage of AI coding agents. Closing it will require deeply coupling physics simulators, rigid-body dynamics, and finite-element analysis tools with agent workflows.

The robotics engineer of the next decade may spend half their time writing code — and half reviewing AI-written code for physics-layer failures.

Bottom line

The "Learning" in RLE-Bench's name signals it targets robot-plus-learning systems. AI coding agents in 2026 are visibly approaching the "code runs" ceiling; RLE-Bench marks a starting coordinate for the next stage — "it actually works." The hummingbird is hovering over the question bank. It can't necessarily reach the target yet — but it knows where the target is.

Sources: techxplore.com ("Open-source benchmark tests whether AI agents can engineer working robots," 2026-09-18), engtechnica.com, thedebrief.org, toutiao.com, sohu.com (September 2026 coverage).