Broken Physics Benchmarks: 47-Point 'Failing' Frontier Models Are Actually Near-90-Point Experts

Forum topic · 小凯 · 2026-09-14

Summary

A Yale, Jump Trading, Cambridge, and USC collaboration re-graded six popular physics benchmarks and found that roughly 95% of frontier-model 'errors' were not model mistakes at all: 57.2% were flawed benchmark questions or wrong reference answers, and 38.0% were grading failures, often from rule-based evaluators that mark mathematically equivalent expressions (e.g., P/(3√6) vs (√6/18)P) as wrong. After expert corrections, GPT-5.6-Sol's scores jumped dramatically: PRISM-Physics 13.0% to 94.6%, PHYBench 26.5% to 90.2%, CritPt 32.3% to 87.5%, CMT-Benchmark 61.0% to 87.2%, and HLE-Physics 47.3% to 78.7%. The paper argues that when a model's true error rate falls below a benchmark's defect rate, reported failures mostly measure the test, not the model. However, near-saturation on closed-form textbook-style problems does not equal research ability: agentic systems still solved no open physics problems. The findings were independently corroborated by Anthropic's corrected CritPt evaluation. The post is a detailed Chinese-language walkthrough of the paper (arXiv:2609.13009), including example defective questions and a critique of benchmark construction.

Broken Physics Benchmarks: A 47-Point 'Failing' Student Is Actually an 87-Point Top Performer

> *"When scores are low, the problem is the student; when scores are compressed toward zero, the problem is the exam itself."*

Paper Overview

Title: *How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks*
Authors: Ali Ansari, Haoran Sun, Andy Zeyi Liu, and 40+ researchers (Yale + Jump Trading Group + Cambridge + USC)
arXiv: 2609.13009

The Anomaly

Popular benchmarks paint frontier models as poor at physics: GPT-5.6-Sol scores only 47.3% on HLE-Physics, 61.0% on CMT-Benchmark, and 32.3% on CritPt. Yet working physicists report frontier models helping reconstruct hidden SL(2,ℝ) symmetry algebras, derive Sudakov resummations of the C parameter, and solve cosmic-string radiation problems analytically. One of these pictures must be wrong — so the authors audited them.

The Audit

40+ physics professors and graduate students re-examined every judged-wrong answer across six benchmarks (250 cases total), classifying each as:

1. Model error — a genuine mistake 2. Grading error — the model was right but the scorer rejected it 3. Benchmark error — the question or reference answer itself is flawed

Results: 95.2% of 'Errors' Are Not the Model's Fault

| Error type | Count | Share | |---|---|---| | Benchmark error (defective question/answer) | 143 | 57.2% | | Grading error | 95 | 38.0% | | Model error | 12 | 4.8% |

Notable per-benchmark findings:

PRISM-Physics: zero model errors among 74 audited cases; after fixes, mean@4 rose from 13.0% to 94.59%
PHYBench: 71.4% of errors were grading errors (e.g., the EED string-matching scorer marking the equivalent form T = P/(3√6) wrong because the reference was (√6/18)P); 26.5% → 90.23%
UGPhysics: 18 of 22 audited items had question defects; only 1 model error

Examples of Broken Questions

Wrong reference answers: an HLE-Physics hyperbolic-function problem where the official answer botched the final arithmetic (−√2 instead of 2−√2); a PHYBench relativity answer that simplifies to zero
Under-specified problems: a PRISM flight-time question with no initial velocity or distance given; the model's honest "insufficient information" was marked wrong
Ambiguous conventions: CritPt #44 on the Kitaev honeycomb model, where using σ vs S = σ/2 changes energies by a factor of four

Key diagnosis: when a model's true error rate falls below a benchmark's defect rate, most reported errors belong to the benchmark, not the model.

Corrected Scores (GPT-5.6-Sol, high reasoning, with tools)

| Benchmark | Pre-audit | Corrected | Gain | |---|---|---|---| | HLE-Physics | 47.3% | 78.7% | +31.4 | | CMT-Benchmark | 61.0% | 87.2% | +26.2 | | CritPt | 32.3% | 87.5% | +55.2 | | PHYBench | 26.5% | 90.2% | +63.7 | | PRISM-Physics | 13.0% | 94.6% | +81.6 | | UGPhysics | 83.0% | 92.1% | +9.1 |

CritPt pass@4 reached 94.4%; CMT pass@4 was 97.96%. Claude Fable 5 rose from 39.5% to 87.6% on PHYBench; Gemini 3.1 Pro from 50.5% to 78.1% on CMT. Anthropic's independent CritPt-Corrected evaluation (31 expert-corrected problems) reported Fable 5.1 at 88.4% mean@1 — an independent replication.

The Caveat: Solving ≠ Research

Near-perfect scores on closed-form, well-defined problems do not mean models can do physics research. The authors ran GPT-driven agentic systems — the kind that helped mathematicians resolve open conjectures — on open theoretical physics problems, and none were solved. Closed-form physics appears near saturation; open-ended research needs new kinds of benchmarks. The paper also notes physics benchmarks lack the executable tests that even SWE-bench (which still showed ~30–60% defective tasks in audits) provides.

Takeaway

1. Reassuring: frontier models' physics reasoning is far stronger than mainstream benchmarks suggest; the "AI is bad at physics" narrative is a measurement artifact. 2. Cautionary: closed-form physics exams are nearly "finished" — the field needs harder, expert-verified, open-problem benchmarks.

In short: the student has outgrown the exam writer's ability to set questions. Next, what needs testing is the question-setting itself.

References

1. Ansari A., Sun H., Liu A.Z., et al. arXiv:2609.13009, 2026. 2. HLE consortium, arXiv:2501.14249, 2026. 3. Pan H., et al., CMT-Benchmark, arXiv:2510.05228, 2026. 4. Zhu M., et al., CritPt, arXiv:2509.26574, 2026. 5. Qiu S., et al., PHYBench, arXiv:2504.16074, 2026. 6. Xu X., et al., UGPhysics, arXiv:2502.00334, 2026. 7. Zhao W., et al., PRISM-Physics, arXiv:2510.03185, 2026. 8. Schwartz M.D., "Vibe physics: the AI grad student," 2026. 9. Anthropic, Claude Fable 5.1 / Mythos 5.1 system card, 2026. 10. Northcutt C.G., Athalye A., Mueller J., arXiv:2103.14749, 2021.