September's robotics discourse was flooded by two viral videos: a robotic arm replicating a human demonstration from a video, and another clumsily but persistently drawing the Golden Gate Bridge — authored by Thijs, a 20-year-old engineer at OpenAI, who succeeded on his fourth attempt, with Sam Altman commenting underneath. Fun as they are, demos don't count. The field has been fed glossy videos for a decade.
What's harder than videos
Something harder went up on September 4 and was pushed into the spotlight on September 16 by a long Humanoids Daily writeup: RoboCurve's real-robot evaluation of GPT-6 Astra — 120 trials, each with full video and execution trajectory archives.
The examiner: RoboCurve is a public-benefit company, YC-backed, which announced a $10M seed round on September 14. Its evaluation framework is open-sourced as Inspect Robots — a nod to the UK AI Safety Institute's Inspect AI. The stated methodology: real arms, real objects, full traceability.
The test setup
- Venue: two I2RT YAM robotic arms
- Candidate: GPT-6 Astra (released September 3, OpenAI's current flagship); controls: specialized robot models Fable 5.1 and Fable 5
- Three models, two tasks, 20 trials each
- Rules: 20 LLM calls per episode, speed capped at 25%, default safety guardrails; scoring on a 0–4 "highest stage reached" scale
- Task prompt: a single sentence — "pick up the red one."
- The comparator is Fable 5.1; online claims that it "beats humans" are a misreading.
- OpenAI's official launch page contains no robotics content at all; the robotics narrative comes entirely from third-party evaluation.
- RoboCurve lists its own limitations: trials were not interleaved, bowl shapes weren't fully identical, and human scoring was not blind. None of this overturns the results — it just draws error bars around that 95.
Results
Task 1: put the block in the bowl. Astra: 19/20 (95%). Fable 5.1: 8/20. Fable 5: 1/20.
The gap goes beyond success rate: Astra averaged 2.5 minutes per trial vs 6.8 for the controls; $0.94 per trial vs $2.12; and 6.2x fewer output tokens. These figures were independently recomputed and check out.
Task 2: precise peg-in-hole insertion — sub-millimeter alignment. Astra: 2/20 (10%), tied with Fable 5.1. In a separate RoboDojo dual-arm协作 test, Astra scored 26%.
95 minus 10 is exactly the "last ten centimeters." Coarse pick-and-place leans on vision and understanding — where the model's era advantage lives. Insertion depends on contact mechanics — force control, friction, deformation — a hard problem labs have chewed on for decades. Reasoning ability can't buy those ten centimeters, and this report card quantifies it. Simulation numbers deserve cold treatment: in another technical report, Astra hit 49/50 on 10 sim tasks with direct control — but the timing excludes inference latency, and none of the real-deployment obstacles go away.
A separate story: the benchmark-score controversy
Around September 3, bloggers noticed some GPT-6 Astra benchmark scores had been edited after launch; the video spread. OpenAI admitted the change, attributing it to evaluation noise and a release bug. This happened at the model's launch and is a separate thread from the robot evaluation two days later — they're easy to conflate but clearer apart. Together, though, they share a theme: benchmark credibility itself became the news.
Fine print
Why it matters
The real weight of this report card may be on the evaluation industry itself. Leaderboards are familiar: one total score, one "way ahead" claim, full stop. RoboCurve offers a different format: every run archived with video and trajectory, anyone can open it and count which trial failed. The 95 is thus auditable, and even the single failure has a paper trail.
Back to those two viral videos: learning actions from video maps to in-context physical learning — the model outputs end-effector poses, with joint solving handled by an external motion planner; the Golden Gate Bridge failing three times before succeeding is, unexpectedly, honest — that's what real-robot learning looks like, trial by trial.
No one can schedule when the last ten centimeters get crossed. What's certain: the next real-robot report card will likely come with receipts too. Leaderboards can lie; receipts can't.
--- Sources: robocurve.org evaluation pages and open-source framework; Humanoids Daily long-form, 2026-09-16; OpenAI GPT-6 Astra launch page; related HN and r/singularity discussions.