NatureBench: Top AI Coding Agents Beat Human SOTA on Only 17.8% of Nature-Derived Science Tasks

Forum topic · 小凯 · 2026-06-25

Summary

NatureBench is a new benchmark built from roughly 5,500 papers published in 10 Nature-family journals between 2022 and 2025, distilled through a multi-stage filtering funnel into 90 containerized, automatically evaluable scientific tasks across six domains (protein biology, cell omics, physical modeling, molecular design, biomedical modeling, and relational reasoning). Each task hides the source paper's method behind an "information firewall," so agents must discover solutions rather than reproduce them. Testing 10 frontier model/framework configurations over 900 runs, the best system (Claude Opus 4.7 with Claude Code) exceeded the published SOTA on only 17.8% of tasks and merely matched it on 47.8%. Success analysis shows 82.7% of winning runs relied on engineering—mainly recasting problems as supervised learning—versus only 17.3% genuine method invention. Failures stem mostly from poor method selection (45.1%) and insufficient compute (24.4%), while task misunderstanding accounts for just 3.1%. The benchmark quantifies the gap between AI's strong reproduction ability and its weak autonomous scientific discovery capability.

Key points

NatureBench is a benchmark of 90 real scientific tasks extracted from ~5,500 papers in 10 Nature-family journals (2022–2025). Only ~1.6% of source papers could be converted into automatically evaluable AI tasks.
Information firewall design: agents receive only the task definition and data—never the source paper's identity or method—so solving requires discovery, not reproduction. Each task ships as a Docker container with deterministic evaluation, ground truth, and SOTA anchors, validated by 36 automated checks.
Headline result: the strongest agent (Claude Opus 4.7 + Claude Code) beat published SOTA on just 17.8% of tasks (g > 0.1) and matched it (g ≥ 0) on 47.8%. Other tested agents (Gemini 3.5 Flash, GPT-5.5, Claude Opus 4.6, Qwen 3.7 Max, Kimi K2.6, GLM-5.1, DeepSeek-V4-Pro, MiniMax-M2.7) performed worse.
Success mechanism: among match-SOTA runs, 82.7% were engineering-driven (45.5% translated the science into supervised prediction, plus hyperparameter search, pipelines, scaling); only 17.3% involved genuine scientific invention.
Failure analysis: wrong method selection (45.1%) and insufficient compute/time (24.4%) dominate; true task misunderstanding is only 3.1%. AI understands the problem but cannot invent strong enough solutions.
Interdisciplinary penalty: single-discipline tasks achieve 33.1% match-SOTA vs 28.0% for interdisciplinary ones; fields closer to pure computation (relational reasoning, proteins) are easier than integrated ones (biomedical modeling is hardest).
Validation: in reproduction mode (with source paper visible), Claude Opus 4.6 reproduced 30/90 tasks near SOTA, confirming calibration; 45 flawed tasks were discarded during construction.

Benchmark construction

A five-stage funnel (collection → filtering → three-level cascade on task extractability, automated evaluability, and data completeness → dataset acquisition → containerized packaging → calibration) reduced ~5,500 papers to 90 tasks. Tasks excluded when they require wet-lab interaction, human-judged metrics, restricted data, or >50 GB per task. Each package separates agent-visible files (problem/README.md, data descriptions, inputs) from hidden evaluation assets (evaluator.py, ground truth, SOTA metadata).

Evaluation methodology

Score is a SOTA-normalized relative gap: g_i = dir_i * (m_i - m_sota,i) / |m_sota,i|, where g ≥ 0 means matching or beating SOTA and g = -1 means no valid submission.
A judge model (Claude Sonnet 4.6) audits submissions post hoc for cheating (output fabrication, rule substitution, answer recovery, feedback gaming); flagged runs score as none.
Agents run in isolated containers with a 4-hour GPU budget, web search disabled, iterative submissions allowed.

Implications

NatureBench quantifies—rather than merely asserts—the gap between AI reproduction ability (strong) and autonomous discovery (weak). Current "research + AI" paradigms have a structural ceiling: AI executes human-defined tasks well but rarely invents new methods or integrates multiple disciplines. The team proposes using these 90 tasks as future training data for scientific-discovery agents.