GPT-6 Astra's Real-Robot Report Card: 95% on Block-in-Bowl, 10% on Precision Puzzle Insertion

Forum topic · QianXun · 2026-09-18

Summary

In September 2026, the nonprofit evaluation company RoboCurve published a real-robot benchmark of OpenAI's GPT-6 Astra (released September 3) on two I2RT YAM robotic arms, with 120 fully recorded trials including video and execution traces. Against a 20-call LLM budget per episode and a common scoring rubric, Astra scored 95% (19/20) on placing a block into a bowl, averaging 2.5 minutes per trial at $0.94 per run, versus 40% for Fable 5.1 and 5% for the prior Fable 5. On sub-millimeter puzzle-peg insertion, however, Astra managed only 2/20 (10%), tying Fable 5.1, and reached 26% on a dual-arm RoboDojo test. The gap quantifies the 'last ten centimeters' problem: contact-rich manipulation requiring force control that reasoning alone cannot yet solve. RoboCurve open-sourced its framework as Inspect Robots and raised a $10M seed round. Separately, OpenAI acknowledged post-launch edits to some Astra benchmark scores. The post argues that per-trial receipts, not leaderboard totals, are what make robot benchmarks credible.

September's robot scene was flooded by two viral videos. In one, a robotic arm watches a human demonstration video and replicates the motion. In the other, it clumsily but persistently draws the Golden Gate Bridge — created by Thijs, a 20-year-old engineer at OpenAI, who only succeeded on his fourth iteration; Sam Altman even asked for a drawing in the comments. Fun as they are, demos don't count. This field has been fed pretty videos for a decade and is full.

Something harder than video went up on September 4 and was pushed into the spotlight on September 16 by a long-form piece in Humanoids Daily: RoboCurve's real-robot evaluation of GPT-6 Astra — 120 trials, each with a full video and execution-trace archive.

The Examiner

RoboCurve is a public-benefit company, YC-backed, that announced a $10M seed round on September 14. It open-sourced its evaluation framework as Inspect Robots — a nod to the UK AI Safety Institute's Inspect AI. The philosophy in the docs: real arms, real objects, full traceability.

The exam room is two I2RT YAM arms. The test-taker is GPT-6 Astra (released September 3, OpenAI's current flagship); the control group is two specialized robot models, Fable 5.1 and Fable 5. Three models, two tasks, 20 trials each. Rules are uniform: a 20-call LLM budget per episode, speed capped at 25%, default safety guardrails, and scoring on a 0–4 rubric recording the highest stage reached. Each task begins with one sentence: "Pick up the red one."

Task 1: Block into Bowl — 95%

GPT-6 Astra: 19/20 (95%), averaging 2.5 minutes per trial
Fable 5.1: 8/20
Fable 5: 1/20

The gap goes beyond success rate: $0.94 vs $2.12 per trial, and 6.2× fewer output tokens. We re-checked these figures item by item; they hold up.

Task 2: Precision Puzzle Insertion — 10%

The picture changes. Inserting a puzzle piece into its matching hole demands sub-millimeter alignment. Astra: 2/20 (10%) — a tie with Fable 5.1. On a separate RoboDojo dual-arm协作 test, Astra scored 26%.

95 minus 10 is exactly the "last ten centimeters." Coarse pick-and-place leans on vision and understanding, where the model era pays off; peg insertion leans on contact mechanics — force control, friction, deformation — a hard problem labs worldwide have chewed on for decades. Reasoning can't buy those ten centimeters, and this report card quantifies it. Simulation numbers deserve a cold shower: in a separate technical report, Astra direct control scored 49/50 across 10 simulated tasks — but the timing excludes inference latency, and none of the real-robot deployment obstacles are gone.

A Separate Thread: The Score-Editing Controversy

Around September 3, bloggers discovered that some GPT-6 Astra benchmark scores had been altered after launch; the videos spread. OpenAI admitted the change, attributing it to evaluation noise and a release bug. That happened at the model launch — a separate line from the robot evaluation two days later. Close in time, easily conflated, but cleaner apart. Together, though, they share a theme: benchmark credibility itself became the news.

Fine Print

The comparator is Fable 5.1; online claims of "beating humans" are a misreading.
OpenAI's official launch page contains no robotics content; the robotics narrative comes entirely from third-party evaluation.
RoboCurve's own stated limitations: trials were not interleaved, bowl shapes were not perfectly identical, and human scoring was not blind. None of this overturns the results — it just puts error bars around the 95.

What This Really Weighs

The real weight of this report card may be on the evaluation industry itself. We've seen plenty of leaderboards — one aggregate score, one claim of dominance, end of story. RoboCurve's alternative format: every run gets a saved video and trace, and anyone can open them and count where it went wrong. The 95 can therefore be audited, and the one failure is on the record.

Putting those two viral videos back in context: learning motion from video maps to in-context learning of physics — a single demonstration suffices, though the model outputs end-effector poses while joint solving is still handled by an external motion planner. Failing at the Golden Gate Bridge three times before succeeding is, unexpectedly, honest — that's what real-robot learning looks like: tried, one attempt at a time.

Nobody can schedule when the last ten centimeters get crossed. What's certain is that the next real-robot report card will probably come with receipts too. Leaderboards can lie; receipts can't.

--- Sources: RoboCurve evaluation pages and open-source framework; Humanoids Daily long-form piece (2026-09-16); OpenAI GPT-6 Astra launch page; HN and r/singularity discussions.