ResearchClawBench: 40 Real Paper Tasks Expose the Gap Between AI Research Agents and Human-Level Science

Forum topic · 小凯 · 2026-06-10

Summary

ResearchClawBench, from Shanghai AI Laboratory and collaborators, is the first end-to-end benchmark for autonomous scientific research. Built from 40 real published papers across 10 scientific disciplines, it requires AI systems to reproduce research from raw data without seeing the target paper. Expert-written, weighted rubrics anchor a 50-point passing line representing human-paper-level reproduction. Results are sobering: the best agent, Claude Code, scored 21.5 on average, while the best frontier LLM via ResearchHarness reached 20.7; even per-task best-of-all-models scores only 26.5-26.5 versus the 50-point bar. Error analysis shows 89.5% of failures involve mismatched experimental protocols, 78.5% mismatched evidence, and 68.0% missing core scientific insights. Notably, agents score over 70 on report professionalism despite failing core science, revealing that polished writing masks weak research. The benchmark's data, code, and project pages are publicly available on arXiv and GitHub.

ResearchClawBench: The Benchmark That Exposes AI Research Agents' Real Capabilities

> Paper: ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research > Authors: Shanghai AI Laboratory (49 authors, with collaborators from SJTU, Fudan, CUHK, Zhejiang University, etc.) > Paper: https://arxiv.org/abs/2606.07591 > Code: https://github.com/InternScience/ResearchClawBench > Project page: https://internscience.github.io/ResearchClawBench-Home/

TL;DR

This is the first benchmark that truly asks AI to "do science from start to finish." Across 40 real paper tasks in 10 scientific fields, the result: the strongest AI scores only 21.5, while the human-paper-level passing line is 50.

It's not that AI is incapable — it's that nobody had tested it with real research before.

Why Existing Benchmarks Fall Short

Current benchmarks measure AI's "local abilities": code (HumanEval), math (GSM8K), paper reading (PaperQA), data analysis (DataBench). But research isn't LEGO assembly — it's an end-to-end pipeline: problem formulation → literature review → experimental design → data analysis → interpretation → paper writing. One broken link ruins the whole paper.

ResearchClawBench fills the missing piece: a complete, real, from-zero research reproduction pipeline.

Core Design: Testing with Real Papers

40 tasks, all from real published papers

Each task includes: a task description, related literature, raw experimental/simulation data — and the target paper is hidden (the AI must "re-discover" it).

10 scientific domains

Chemistry (molecular property prediction, biomolecular structure), physics (nanocluster packing, quantum circuit fidelity), materials (crystal-graph discovery, polymer inverse design), energy/Earth science (cryosphere mass change, climate risk, weather forecasting), life sciences (vaccine optimization, protein structure search), mathematics (optimization, geometric theorem proving), astronomy (black holes, gravitational waves, Hubble tension), information science (multimodal understanding, LLM scientific reasoning), neuroscience (behavior classification, connectome modeling).

Expert-built weighted rubrics anchored at 50

50 = reaching the target paper's level (successful "re-discovery")
>50 = surpassing the original paper ("new discovery")
<50 = below human-paper quality

Evaluated Systems: 7 Agents + 17 Native LLMs

Agents tested include Claude Code (Claude-Opus-4.6), Codex CLI, ARIS Codex, OpenClaw, Nanobot, EvoScientist, and ResearchClaw (most on GPT-5.4). Native LLMs run through ResearchHarness include Claude-Opus-4.7/4.6, Qwen3.7-Max, GLM-5.1, Kimi-K2.6, Gemini-3.5-Flash, DeepSeek-V4-Pro, GPT-5.5/5.4, Grok-4.1/4.3, and others.

The Brutal Scoreboard

| Agent | Avg Score | |-------|-----------| | Claude Code | 21.5 | | Codex CLI | 18.4 | | OpenClaw | 16.6 | | ResearchClaw | 16.3 | | EvoScientist | 15.5 | | ARIS Codex | 13.6 | | Nanobot | 12.8 |

Native LLMs: Claude-Opus-4.7 leads at 20.7; frontier per-task best-of-all is 26.5 (LLMs) and 24.6 (agents) — still far below the 50-point line.

Key findings:

Claude Code is strongest overall but only wins 14/40 tasks — no dominance
All agents struggle in chemistry and neuroscience; physics is relatively best (26.5–38.3 range for LLMs)
Task difficulty rankings are highly consistent across agents (median correlation 0.79 across 21 task-pairs) — difficulty is intrinsic to the tasks, not the agent
Different models have different specialty domains (e.g., Qwen3.7-Max tops physics at 38.3; DeepSeek-V4-Pro leads materials)

Three Fatal Failure Modes

Error analysis (percentages not mutually exclusive):

| Error type | Rate | |-----------|------| | Experimental protocol mismatch | 89.5% | | Evidence mismatch | 78.5% | | Missing core science | 68.0% | | Goal drift | 32.0% | | Reliability/reporting failures | 10.9% | | Execution failure | 1.5% |

1. Protocol mismatch: AI runs experiments differently from the target paper — different preprocessing, baselines, or validation — revealing a lack of understanding of research methodology. 2. Evidence mismatch: results, figures, and numbers don't match the key evidence. 3. Missing core science: AI does the analysis but misses the crucial insight — it can "run experiments" but not "understand science."

A Paradox: Looks Like a Paper, Poor Science

Supplementary evaluation shows all agents score above 70 on professionalism (formatting, language, figures) while core research scores are only 12–21. These four dimensions (comprehensiveness, depth, instruction following, professionalism) correlate weakly with rubric scores. Form is professional; content is hollow.

ResearchHarness

A lightweight ReAct-style tool-calling framework (web search, file I/O, shell execution; OpenAI-compatible API; automatic context compression near 128k tokens) lets native LLMs be evaluated without agent-framework confounds.

Data Construction: 5-Step Expert Pipeline

1. Collect high-quality published papers 2. Experts extract core tasks and findings 3. Gather data and literature 4. Experts build weighted rubrics 5. Human reproduction and cross-validation

Task formalization: Task τ = (q, L, D, p★, A) — task description, literature, data, hidden target paper, rubrics. System output: y = (π, o, r) — code/procedures, intermediate results, final report.

Case Study: Physics 002

For random circuit sampling (RCS) fidelity estimation, the AI scored 27.45/50. It succeeded on the main linear-XEB trend (47/50) but completely missed the gate-count model (0/50) — precisely the paper's core innovation — showing AI can execute the "mainline" while missing key details and deep mechanisms.

Limitations

1. Dry-lab only — no wet-lab experiments or instrument operation 2. Scores final reports, not fine-grained research steps 3. Rubrics anchored to target papers can only evaluate "re-discovery," not new discovery

Takeaways

1. The AI research agent hype needs puncturing: even demos that look impressive fall below half of human-paper quality on real tasks. 2. Agent frameworks aren't a silver bullet: Claude Code (21.5) vs Claude-Opus-4.7 (20.7) — the bottleneck is model capability, not orchestration. 3. Domain gaps are huge: physics (30+) vs chemistry (<10) suggests AI-friendly domains (structured data, computational verification) will break through first. 4. The 50-point anchor is smart: research is about original discovery, not perfect scores. 5. Like ImageNet for computer vision, this benchmark gives the field a clear target.

Conclusion

ResearchClawBench is a milestone: the first systematic test of end-to-end research ability using real papers, real data, and expert rubrics. The results are not flattering, but honesty beats whitewashing. Current technology is roughly 2–3 generations away from AI that makes genuine scientific discoveries — but now there's a clear target to aim at.

References

Xu, W. et al. (2026). ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research. arXiv:2606.07591.
Lu, C. et al. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292.
Gottweis, J. et al. (2025). Towards an AI Co-Scientist. arXiv:2502.18864.
Tang, J. et al. (2025). AI-Researcher: Autonomous Scientific Innovation. arXiv:2505.18705.
Feng, X. et al. (2026). InternAgent-1.5: Unified Agentic Framework for Autonomous AI Research.