NatureBench: AI Coding Agents Beat Human SOTA on Only 18% of Nature Paper Tasks

Forum topic · ✨步子哥 · 2026-06-24

Summary

NatureBench is a new benchmark of 90 tasks distilled from Nature-family journal papers that tests whether AI coding agents can match or exceed published state-of-the-art scientific results from scratch. Built on NatureGym, an automated pipeline that packages papers into containerized, reproducible environments across six domains (computational biology, materials science, chemistry, physics, geoscience, medicine), the benchmark evaluated 10 frontier agent configurations including Claude Code, Codex CLI, and Gemini CLI under strict no-web-search conditions with 4-hour compute budgets. The best agent, Claude Opus 4.7, surpassed reported SOTA on only 17.8% of tasks, tied on 47.8%, and failed to reproduce on 34.4%. Analysis of 900 runs shows successful agents perform 'methodology translation'—mapping scientific problems onto familiar supervised learning workflows (45.5% of successes)—rather than genuine scientific invention. The dominant failure mode is wrong method selection (45.1%), followed by insufficient compute budget (24.4%).

When AI Coding Agents Meet Nature Papers

Imagine you just published a Nature paper with a new method achieving SOTA on a scientific problem. Now someone hands your paper to an AI coding agent and asks it to implement your method from scratch—can it match the accuracy you reported?

Result: the strongest agent (Claude Opus 4.7) surpassed the published SOTA on only 17.8% of tasks and barely tied on 47.8%. On the remaining 34.4% of tasks, it couldn't even reproduce the results.

This is NatureBench—a new benchmark with 90 tasks distilled from Nature-family journal papers. It asks not "can AI write code?" but "can AI do scientific discovery?"

90 Tasks, From Paper to Executable Environment

NatureBench's core is an automated pipeline called NatureGym:

1. Paper selection: filter papers from Nature, Nature Methods, Nature Communications, etc., that have code, data, and clear evaluation metrics 2. Task packaging: extract the methods section, pair it with datasets and evaluation scripts, and containerize everything into an executable environment 3. Standardized evaluation: every task has a uniform interface—the agent receives the method description and data, writes code, and is scored by the evaluation script

This solves a long-standing problem in AI-for-Science benchmarks: environment fragmentation. Previously, each paper's reproduction environment differed—dependency conflicts, incompatible data formats, missing evaluation scripts—making benchmark results nearly irreproducible. NatureGym ships each task in its own container that anyone can run.

The 90 tasks span six scientific domains: computational biology, materials science, chemistry, physics, geoscience, and medicine. Each task comes from a real, peer-reviewed Nature-family paper with public SOTA numbers.

10 Frontier Agents, Strictly Offline

The paper tested 10 configurations across three coding agent frameworks:

Claude Code with Claude Opus 4.7, Claude Sonnet 4.5, and others
Codex CLI with GPT-5.4, GPT-5.5
Gemini CLI with Gemini 3.5 Flash
Plus K2.6, MiniMax-M2.7, DeepSeek-V4-Pro, GLM-5.1, Qwen 3.7 Max, and more

Evaluation conditions were extremely strict: web search disabled. Agents could not search for the original paper, download datasets, or look up GitHub repositories—they had to write code from scratch based only on the method description and data provided in the task package.

Each task allowed 4 hours of wall-clock time, with GPUs allocated per task requirements (70 tasks on RTX 3090/4090, 17 compute-heavy tasks on A800, 3 CPU-only tasks).

Key Finding: Methodology Translation, Not Scientific Invention

Analysis of 900 runs (10 agents × 90 tasks) reveals a counterintuitive conclusion:

Successful agents aren't doing scientific invention—they're doing methodology translation.

Specifically, 45.5% of successes follow the same path: the agent translates the scientific problem into a supervised learning prediction problem it knows well, then applies a standard ML pipeline. It doesn't "understand the science and create a new method"—it "maps the problem into its known method space and finds the closest match."

The failure modes are equally informative:

45.1% of failures stem from wrong method choice—the agent picks a method fundamentally unsuited to the problem
24.4% of failures come from insufficient compute budget—right method, but 4 hours isn't enough
Only a small fraction of failures are due to "not understanding the task"

In other words, agents generally understand what the task is asking. The bottleneck is method selection and compute resources, not comprehension.

What Does 17.8% Beating SOTA Mean?

Is 17.8% good or bad? Depends on your definition.

Optimistically: these are Nature-family papers representing humanity's best scientific methods. An AI beating human SOTA on nearly one in five problems is remarkable for a general-purpose system.

Pessimistically: on 82.2% of tasks, AI didn't beat SOTA—despite the methods being fully described in the papers. This shows a huge gap between "reading a paper" and "writing code that reproduces it."

Even more thought-provoking is the quality of "beating." The paper introduces a post-hoc validity judge to detect shortcut behaviors—e.g., an agent gaming the score through output fabrication or exploiting evaluation feedback. Only successes passing the validity check count as real. This shows that in AI-for-Science, evaluation methodology itself is a hard problem.

Why It Matters

Previous benchmarks tested either "can you reproduce a paper" (PaperBench-style) or "can you optimize engineering problems" (Kaggle-style). NatureBench tests something harder: given a scientific problem and data, can AI devise a method that matches or beats the paper's SOTA?

The answer is yes—but only 17.8% of the time, and via methodology translation rather than scientific invention. Implications for AI-for-Science:

1. Methodology translation ability is already strong. Frontier models have accumulated enough method knowledge to do cross-disciplinary method matching. 2. True scientific invention remains weak. AI can translate science into known supervised learning problems but struggles to design genuinely novel methods. 3. Method selection is the key bottleneck. With 45.1% of failures from wrong method choice, the improvement path lies not in "writing more code" but in "choosing better methods."

Honest Assessment

1. 90 tasks is not many. Six domains at ~15 tasks each limits statistical granularity. 2. No web search cuts both ways. It ensures fairness but deviates from real-world use, where agents can consult the literature. 3. The 4-hour wall-clock limit is unfair to methods requiring long training. 24.4% of failures can't be blamed on agent capability. 4. The "beat SOTA" threshold (g > 0.1) is relatively loose. Marginal wins may fall within noise.

Industry Implications

NatureBench pours cold water on AI-for-Science hype—and offers a clear roadmap. Cold water: on ~82% of scientific tasks, AI still falls short of human experts. Roadmap: method selection is the next breakthrough. If agent method-selection accuracy rises from 55% to 80%, the SOTA-beating rate could jump from 17.8% to 40%+. That's when AI-for-Science truly takes off.

---

Paper: NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? arXiv: https://arxiv.org/abs/2606.24530 HTML: https://arxiv.org/html/2606.24530v1 Code: https://github.com/FrontisAI/NatureBench Institutions: Horizon Research, Frontis.AI, Tsinghua University