NatureBench: Throwing AI Into Nature-Level Labs — It Can't Even Beat Half the Tasks

Forum topic · 小凯 · 2026-06-25

Summary

NatureBench is a benchmark of 90 real scientific tasks distilled from roughly 5,500 papers published in 10 Nature-family journals (2022-2025), designed to test whether AI can make genuine scientific discoveries rather than reproduce existing methods. An information firewall strips away the source paper's methods so the AI sees only task definitions and data. Results are sobering: the best coding agent tested, Claude Opus 4.7, surpassed human state-of-the-art (SOTA) on only 17.8% of tasks and merely matched it on 47.8%. Interdisciplinary tasks proved especially hard. Analysis of 900 runs shows AI succeeds not through scientific invention but by translating problems into standard supervised learning — engineering-driven strategies account for 82.7% of successes versus 17.3% method invention. Failures stem mainly from wrong method choice (45.1%) and insufficient compute (24.4%), not task misunderstanding (3.1%). The benchmark quantifies the qualitative gap between reproduction and discovery in AI for Science.

NatureBench: Can AI Do Nature-Level Scientific Discovery?

NatureBench is a benchmark built from ~5,500 papers across 10 Nature-family journals (2022-2025), filtered down to 90 automatable scientific tasks — only about 1.6% of the initial pool qualified. Its core question: can AI *discover* methods that beat published SOTA, rather than reproduce them?

Why NatureBench?

Existing benchmarks test the wrong thing:

PaperBench-style: reproducing paper code — "copying," not discovering
MLE-bench-style: Kaggle engineering optimization, not scientific exploration
Fragmented evaluation environments make comparison difficult

The Filtering Funnel

| Stage | Papers remaining | |-------|------------------| | Collected from Nature journals | ~5,500 | | After basic filtering | ~2,500 | | Three-level cascade (extractability, auto-evaluability, data completeness) | ~200 | | After dataset acquisition | ~180 | | Task package construction | ~160 | | After calibration/audit | 90 |

Most papers can't be gamified: they need wet labs, human judgment, restricted data, or measure speed/cost rather than accuracy. Task domains include relation reasoning (easiest, 60% Match-SOTA), protein biology, cell omics, physics modeling, molecular design, and biomedical modeling (hardest, 17.9%).

NatureGym and the Information Firewall

Each task ships as a Docker container. Crucially, an information firewall removes the source method: the agent sees only the problem README, data description, and input data — never the evaluator, ground truth, or source paper identity. 36 automated checks validate each package.

Evaluation

Performance uses a SOTA-normalized relative gap:

g_i = \text{dir}_i \cdot \frac{m_i - m_{\text{sota},i}}{|m_{\text{sota},i}|}

where g ≥ 0 means matching or beating SOTA. A Claude Sonnet 4.6-based judge reviews submissions for cheating (output fabrication, rule substitution, answer recovery). Web search is disabled; each run gets a 4-hour GPU budget.

Results: 900 Runs Across 10 Agents

| Model | Beats SOTA (g>0.1) | Matches SOTA (g≥0) | |-------|---------------------|---------------------| | Claude Opus 4.7 | 17.8% | 47.8% | | Gemini 3.5 Flash | 15.6% | 37.8% | | GPT-5.5 | 14.4% | 44.4% | | Claude Opus 4.6 | 12.2% | 36.7% | | Qwen 3.7 Max | 10.0% | 28.9% |

Even the best agent beat SOTA on fewer than 1 in 5 tasks. Interdisciplinary tasks (median g = -0.21) were harder than single-domain ones (-0.13).

How AI Succeeds — and Fails

Across 290 Match-SOTA runs:

82.7% of successes were engineering-driven; only 17.3% involved method invention
45.5% of successes came from recasting problems as supervised prediction
Runs aligned with the source paper's method family matched SOTA 37.7% of the time vs. 29.6% for alternative families

Across 610 failures:

45.1%: wrong method choice (method layer)
24.4%: insufficient compute/time
Only 3.1%: task misunderstanding

AI understands the problem — it just can't invent a good enough solution. In reproduction mode, Claude Opus 4.6 could reproduce 30/90 tasks, confirming the benchmark's SOTA anchors are well calibrated.

Key Takeaways

17.8% — top AI surpasses human SOTA on under a fifth of Nature-level tasks
82.7% vs 17.3% — AI "success" is engineering optimization, not scientific invention
45.1% vs 3.1% — failures come from method creativity, not comprehension
The gap between reproduction and discovery is qualitative, not incremental

The authors' long-term goal is to turn the benchmark into training data for future scientific-discovery AI. That day, they note, has not yet arrived.

*The code and benchmark are slated for open-source release per the paper's updates.*