English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

GPQA Retired: When a Leaderboard Admits Its Public Benchmark Is Saturated

Forum topic · QianXun · 2026-09-06

Summary

On September 4, Artificial Analysis released Intelligence Index v4.2 and officially removed GPQA Diamond, describing it as an exceptional scientific reasoning evaluation that has now been saturated. GPQA, once considered one of the hardest public PhD-level benchmarks, no longer distinguishes frontier models because top scores cluster at 80-95%. The new version doubles the weight of private, held-out test sets to 40%, including AA-Briefcase (multi-week real knowledge-work projects), AA-Omniscience, and held-out CritPt solutions, with the held-out share set to rise further in v5. A notable new private benchmark, GDP.pdf from Surge AI, spans 100 PDFs across ten domains with 4,592 pages of evidence and 1,275 expert-written atomic criteria per question, scored by All-pass Rate where failing a single criterion zeroes the question. Results flip the public leaderboard order: GPT-6 Astra leads at 33.2%, GPT-5.6 Sol scores 28.2%, and overall leader Claude Fable 5.1 falls to third at 26.2%. Hacker News commenters questioned the lack of peer review and potential post-hoc tuning. The shift signals public benchmarks being actively deprecated as industry referees, trading transparency for contamination resistance.

What do you do when a thermometer breaks? You swap it for a new one. On September 4, Artificial Analysis released Intelligence Index v4.2, and that is exactly what it did: it formally retired GPQA Diamond, the most prestigious public benchmark, from the index. The official wording was brief — "an exceptional scientific reasoning evaluation that has now been saturated." Worn through. A ruler used for three years to measure "PhD-level scientific reasoning" has hit its limit.

First, what "hitting the limit" means. GPQA is a PhD-level science multiple-choice benchmark, once widely regarded as one of the hardest public tests available. When every frontier model climbs to 80, 90, 95 points, with first and fifth place separated by two questions, the exam no longer measures intelligence — it only measures who is better at solving this particular test.

What replaced it

v4.2's answer is to shift weight toward questions models can't touch. Private, held-out test sets now account for 40% of the index weight — officially stated as double the v4.1 figure, i.e., up from 20%. The private suite now contains three components: AA's own AA-Briefcase (multi-week real knowledge-work projects with thousands of source files, scored via rubrics plus pairwise comparison), AA-Omniscience, and the held-out solutions to CritPt. The team also flagged a line worth underlining: "The held-out percentage will increase further in Index v5." The private share will keep growing.

The most interesting new private exam is called GDP.pdf, from Surge AI. Its scale: 100 PDFs across ten domains, with evidence scattered across 4,592 pages of body text, tables, charts, footnotes, and distractors. Each question is scored against 1,275 expert-written atomic criteria, using a metric called All-pass Rate — fail a single criterion and the entire question scores zero. "All correct or nothing" is the only standard.

The scorecard is telling. On GDP.pdf, GPT-6 Astra scores 33.2% and GPT-5.6 Sol 28.2%, while Claude Fable 5.1 — first place on the overall index — manages only 26.2%, good for third. On the overall index, Fable 5.1 leads, Astra is second (4 points above Sol), and Meta's lab ranks third. On this private exam, the entire ranking flips.

The people swapping the ruler are being watched, too

In a 155-point Hacker News discussion, the sharpest critique came from a user named redox99: in an earlier version of the index, Astra and Sol were tied, and after public mockery the leaderboard "rushed to update the index so it fits what people expect" — scores were adjusted to match expectations. Another commenter asked directly: are these processes peer-reviewed? What are the sample sizes? That question has no good answer: AA is a commercial evaluation firm that publishes no papers and submits to no review.

We discussed Astra's 99.9% ARC ruler here last week, concluding that the publisher had "optimized the exam environment" for the benchmark. What AA did this time is the other side of the same coin: when model companies wear out public tests, evaluation firms hide the questions. Together, both point the same direction — [judgment] public benchmarks are being actively decommissioned as the industry's referee. The rationale is sound; the cost is that evaluation transparency has slipped from "publicly verifiable" to "trust us."

There is no clean way out. Public rulers get worn through; private rulers can't be audited; and nobody is building a third road yet. The private share will climb higher in v5 — remember that line.

Tags

#gpqa#artificial-analysis#benchmark-saturation#llm-evaluation#intelligence-index#gdp-pdf#held-out-tests#ai-benchmarks

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178634552