Put Yourself in That Room
Imagine a house you've never entered, with one instruction: pass through the foyer and stop beside the second floor lamp, to the left of the living room sofa.
You have only a monocular camera — no depth map, no pre-built map, no waypoint predictor trained for navigation. When to look, where to go, when to stop: all your calls.
On September 24, 2026, a paper appeared on arXiv claiming a general-purpose large model achieved a 79.0% success rate on this task (arXiv:2609.29861). Another paper by the same authors, testing the same model with a different workflow, reported 52.0% (arXiv:2609.20116).
Both numbers come from the same researchers, less than a month apart.
The 79% Paper
Authors: Guangzhao Dai and Qianru Sun (Singapore Management University), Qi Wu (Australian Institute for Machine Learning), and Bin Zhu. The approach is minimal: a minimal interface in a Codex harness feeds monocular RGB frames to GPT-6 Astra, letting it decide when to observe, how to move, and when to stop. No navigation fine-tuning, no trained waypoint predictor, no pre-built scene graphs.
Evaluation uses a 100-episode subset of R2R-CE, a common protocol for zero-shot navigation. Success rate is 79.0% at ultra reasoning effort and 76.0% at medium — respectively 13.0 points above the strongest reported zero-shot results and 6.9 points above the strongest supervised results, per the paper.
The paper lists four findings, the third being the most important: reliable path execution and goal confirmation remain difficult even with ultra reasoning enabled. Local landmark matching can look convincing without leading to the correct final endpoint.
The 52% Paper
The second paper takes a different approach: the authors implement their own observe-decide-execute workflow, calling the model API directly — no packaged agent harness, no navigation fine-tuning. External code selects context for each request, retains history of observations and judgments, schedules re-checks, and constrains execution. Evaluated on 50 of the 100 R2R-CE val-unseen episodes.
Results: 52.0% success rate, 48.9% SPL, 70.8% nDTW.
A finer detail on the composition of success: at termination, 36.0% of episodes succeeded via workflow-accepted STOP actions, while another 16.0% were credited at the step limit by satisfying a distance criterion.
In other words, nearly a third of the "successes" in this paper are cases where the model got close but never actively stopped — rescued by the distance criterion.
Where the Difference Lies
Neither paper hides the discrepancy; both lay out their methodology openly. Four layers of difference can be identified:
1. Who wrote the harness — the minimal Codex interface vs. the researchers' own workflow 2. Reasoning effort — 79.0% at ultra; the 52.0% paper does not indicate a comparable setting 3. Evaluation subset — 100 episodes vs. 50 4. Who owns the stop decision — the model itself vs. external code accepting the STOP
The first paper also notes: supervised baselines use the full val-unseen split, so that comparison is not strictly fair.
Most of the 27-point gap is not generated inside the model — it is generated in the code outside the model.
The Same Week, It Drove a Real Car
DrivingBench ran a bolder test: several frontier models directly controlled a 2022 Toyota Corolla along a cone-marked route in an empty parking lot. The car connected to the CAN bus via a comma four device, with openpilot handling low-level control; camera frames and state were passed to models via MCP tools. A human operator sat in the driver's seat with a foot near the brake, speed capped at 0.5–3.5 m/s, with control revoked above 6 m/s.
According to a Chinese tech media report, GPT-6 Astra — running via Codex at medium reasoning effort — was the only model to complete the full course. The first attempt covered 67.3 meters for 49% track progress before termination; without clearing the conversation, the team let it review the failure and retry. The second attempt covered 134.7 meters in 5 minutes 22 seconds for 100% track progress. Among the control group, Claude Fable 5.1's best was 45%, Grok 4.6 got 11%, and GPT-5.6 Sol got 6%. *(Retold via NetEase Yun Toutiao, 2026-09-24, plus the DrivingBench leaderboard; no first-hand paper seen.)*
Three details matter more than "it finished":
1. Cost: the second attempt consumed ~6.6 million tokens, roughly $7.74 at public prices. Spread over 134.7 meters, that is about $92 per mile in model inference alone — before the car, network, and equipment. 2. Speed: 134.7 meters in 5:22 averages ~0.42 m/s, under 1 mph. 3. It initially refused to drive: several models reportedly refused to continue once they realized they were controlling a real car, Astra most visibly. The team renamed the MCP service to DrivingBenchSandbox and modified prompts before it would reliably execute. Even after seeing real camera feeds, it sometimes re-concluded it was driving a real car and stopped.
Three Boundaries
1. Sample size: the 52% paper evaluates 50 episodes, the 79% paper 100 — small enough that a single episode swings several points. Neither paper reports variance, so we cannot tell whether the between-paper gap matches the within-paper confidence intervals. 2. Source grade: the real-car test currently exists only as media retelling and a leaderboard, with no first-hand paper or full protocol. Each model got one full evaluation with up to three attempts, and later attempts retained chat context — so attempts are not independent. 3. "Completed the course" vs. "ready for the road": an entire autonomous-driving engineering stack sits between the two. The retelling is explicit: front camera only — no left, right, or rear view — and openpilot plus the vehicle itself limit maximum steering angle.
Watch List
- Variance reporting: both papers report only point estimates; will follow-up work include multiple runs and confidence intervals?
- Harness openness: the 79% paper relies on internal Codex harness interfaces — how hard will third-party reproduction be?
- Stop-decision ownership: STOP by the model vs. acceptance by external code may split into two evaluation conventions.
- First-hand paper on the real-car experiment: if DrivingBench publishes formally, the protocol and failure cases deserve a full read.
- Refusal behavior: when a model declines to operate real equipment on safety grounds — feature or obstacle, and how should engineering handle it?
References
1. Guangzhao Dai, Qianru Sun, Qi Wu, Bin Zhu, *GPT-6-Astra Lights Up Embodied Navigation: Evaluation in Zero-Shot Vision-and-Language Navigation in Continuous Environments*, arXiv:2609.29861, 2026-09-24 2. Guangzhao Dai, Qi Wu, Bin Zhu, *How Far Can GPT-6-Astra Go? Evaluating capabilities in Zero-Shot Vision-and-Language Navigation*, arXiv:2609.20116 3. NetEase Yun Toutiao retelling of DrivingBench's real-car test of GPT-6 Astra, 2026-09-24 (secondary source; no first-hand paper seen) 4. FutureX Physical AI Daily, Issue 131, 2026-09-26