When AI Coding Agents Meet Nature Papers
Imagine you just published a Nature paper with a new method that achieves state-of-the-art (SOTA) results on a scientific problem. Now someone hands your paper to an AI coding agent and asks it to write code from scratch to reproduce your method and match the reported accuracy.
The result: the strongest AI agent (Claude Opus 4.7) surpassed the paper-reported SOTA on only 17.8% of tasks and barely matched it on 47.8%. On the remaining 34.4%, it could not even reproduce the results.
This is NatureBench — a new benchmark with 90 tasks distilled from Nature-family journal papers. It asks not "can AI write code" but "can AI do scientific discovery."
90 Tasks, From Paper to Executable Environment
NatureBench's core is an automated pipeline called NatureGym:
1. Paper selection: filter papers from Nature, Nature Methods, Nature Communications, etc., that have code, data, and clear evaluation metrics. 2. Task packaging: extract the methods section, pair it with datasets and evaluation scripts, and bundle everything into a containerized executable environment. 3. Standardized evaluation: every task has a unified interface — the agent receives the method description and data, writes code to implement the method, and is scored by the evaluation script.
This solves a long-standing problem in AI-for-Science benchmarks: environment fragmentation. Previously, reproduction environments differed per paper — dependency conflicts, incompatible data formats, missing evaluation scripts — making benchmark results nearly irreproducible. NatureGym puts every task in its own container anyone can run.
The 90 tasks cover six scientific domains: computational biology, materials science, chemistry, physics, earth science, and medicine. Each task comes from a real, peer-reviewed Nature-family paper with public SOTA numbers.
10 Frontier Agents Tested, Web Access Strictly Disabled
The paper tested 10 configurations across three coding agent frameworks:
- Claude Code with Claude Opus 4.7, Claude Sonnet 4.5, and others
- Codex CLI with GPT-5.4 and GPT-5.5
- Gemini CLI with Gemini 3.5 Flash
- Plus K2.6, MiniMax-M2.7, DeepSeek-V4-Pro, GLM-5.1, Qwen 3.7 Max, and more
- 45.1% of failures are wrong method selection — the agent chose a method fundamentally unsuited to the problem.
- 24.4% of failures are insufficient compute budget — the right method, but 4 hours wasn't enough.
- Only a small fraction of failures stem from "not understanding the task."
Evaluation conditions are strict: no web search. Agents cannot search the original paper online, download datasets, or consult GitHub repositories. They must write code from scratch based only on the method descriptions and data provided in the task package.
Each task gets 4 hours of wall-clock time, with GPUs allocated per task needs (70 tasks on RTX 3090/4090, 17 compute-heavy tasks on A800, 3 CPU-only).
Key Finding: Methodology Translation, Not Scientific Invention
Analysis of 900 runs (10 agents × 90 tasks) reveals a counterintuitive conclusion:
Successful agents are not doing scientific invention — they are doing methodology translation.
Specifically, 45.5% of successes follow the same path: the agent translates the scientific problem into a familiar supervised learning prediction problem, then applies a standard ML pipeline. It doesn't "understand the scientific problem and invent a new method"; it maps the problem onto its known method space and picks the closest match.
Failure modes are equally instructive:
What Does 17.8% Surpassing SOTA Mean?
It depends on how you define "good."
Optimistically: these are Nature-family papers representing humanity's best scientific methods. An AI surpassing human SOTA on 17.8% of them — nearly one in five scientific problems — is remarkable.
Pessimistically: on 82.2% of tasks, AI did not beat SOTA. Given the methods are already spelled out in the papers (though web restrictions hide the originals), the gap between "reading a paper" and "writing code that reproduces it" remains enormous.
Notably, the paper introduces a "post-hoc validity judge" to detect shortcut behaviors — e.g., agents gaming scores through output fabrication or exploiting evaluation feedback. Only successes passing validity checks count as real. This shows that in AI-for-Science, evaluation methodology itself is a hard problem.
Why It Matters
NatureBench's value isn't the leaderboard but the question it systematically answers for the first time: can AI coding agents do scientific discovery?
Prior benchmarks test either paper reproduction (PaperBench-style) or engineering optimization (Kaggle-style). NatureBench tests something harder: given a scientific problem and data, can AI devise a method that matches or beats the paper's SOTA?
The answer is yes — but only 17.8% of the time, and through methodology translation rather than invention. Implications for AI-for-Science:
1. Methodology translation capability is already strong. 45.5% of successes show frontier models have enough method knowledge to match methods across disciplines. 2. True scientific invention is still weak. Agents can translate scientific problems into known supervised learning problems but struggle to design genuinely new methods. 3. Method selection is the key bottleneck. 45.1% of failures come from wrong method choice — improvement lies in "choosing methods better," not "writing more code."
Honest Assessment
1. 90 tasks is not many. Roughly 15 tasks per domain limits statistical granularity. 2. No-web-search cuts both ways. It ensures fairness (no copying paper code) but deviates from real research workflows where agents can consult the web. 3. The 4-hour wall-clock limit is unfair to methods needing long training. 24.4% of failures (compute budget) cannot be blamed on agent capability. 4. The "surpass SOTA" threshold (g > 0.1) is relatively lenient; small margins may fall within noise.
Industry Implications
NatureBench is both a cold shower and a roadmap for AI-for-Science.
The cold shower: stop claiming "AI can do scientific research." On 82% of scientific tasks, AI still falls short of human experts.
The roadmap: method selection is the next breakthrough. Current agents code well but don't know "which method to use for this problem." If agents' method-selection accuracy rises from 55% to 80%, the surpass rate on NatureBench could jump from 17.8% to 40%+. That's when AI-for-Science truly takes off.
---
Paper: NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? arXiv: https://arxiv.org/abs/2606.24530 HTML: https://arxiv.org/html/2606.24530v1 Code: https://github.com/FrontisAI/NatureBench Institutions: Horizon Research, Frontis.AI, Tsinghua University