English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

AI Can Write Papers but Can't Judge Them: A 250-Paper Survey of Autonomous AI Research

Forum topic · 小凯 · 2026-05-20

Summary

A comprehensive survey from researchers at the National University of Singapore, CUHK, CNRS, and other institutions reviews 250+ papers on AI-driven automatic research across the full research lifecycle, organized into four phases (creation, writing, validation, dissemination) and eight steps. Key findings: AI systems can now generate complete papers for as little as $15 each (The AI Scientist) and run pipelines producing 100 papers in 228 hours (FARS), but they struggle with substance. LLMs achieve only 37-39% success on research-grade code versus ~80% on pattern-matching code tasks; LLM reviewers misclassify 95.8% of reject-worthy papers as acceptable; and LLM-judged novelty negatively correlates with real-world impact. The survey concludes that generation consistently outpaces verification, and that human-governed human-AI collaboration - not full autonomy - is currently the most reliable deployment mode, with AI handling retrieval, drafting, and visualization while humans retain judgment, experimental design, and accountability.

AI Can Write Papers but Can't Tell If It's Making Things Up: A State-of-the-Field Report

A large-scale survey of AI-driven automatic research reviews 250+ papers (up to April 2026) across the entire research lifecycle, from idea generation through social-media dissemination. Its diagnosis is clear: AI has become strong at *producing the forms of research* but remains weak at *guaranteeing the substance*.

Links:

  • arXiv: https://arxiv.org/abs/2605.18661
  • Project page: https://worldbench.github.io/awesome-ai-auto-research
  • Hugging Face: https://huggingface.co/papers/2605.18661
  • The AI Scientist (Nature): https://www.nature.com/articles/s41586-024-07958-1
  • OpenScholar: https://github.com/OpenScholar
  • ResearchTown: https://github.com/ulab-uiuc/research-town
  • Key Points

    The framework: 4 phases, 8 steps

    The survey unifies AI research automation into an industrial pipeline: Creation (idea generation, literature review, coding & experiments, tables & figures) → Writing → Validation (peer review, rebuttal & revision) → Dissemination (Paper2X: posters, slides, videos, social media, interactive agents). Errors amplify downstream: a missed prior work at the idea stage becomes a false novelty claim at the writing stage.

    Stage-by-stage findings

  • Idea generation: LLM ideas look novel but suffer from novelty hallucination and a feasibility cliff. HindSight found LLM-assessed novelty is *negatively* correlated with real impact (ρ = -0.29). IdeaBench: novelty scores >0.6, feasibility <0.5.
  • Literature review: the most mature stage. OpenScholar (45M papers) outperforms GPT-4o by 6.1%; citation hallucination persists but is addressable (CiteLLM).
  • Coding & experiments: the most sobering result. Research-grade code success is only 37-39% (ResearchCodeBench) vs. ~80% on pattern-matching tasks (SWE-Bench) — a 41-43 point gap. LLMs can fix code but struggle to implement genuinely novel algorithms.
  • Writing: fully automated papers cost ~$15 each (The AI Scientist); Agent Laboratory cut costs 84%; CycleResearcher scores 5.36, close to preprints (5.24) but below accepted papers (5.69). Fluent prose can contain unsupported claims.
  • Peer review: the most alarming number — LLM reviewers misjudge 95.8% of papers that should be rejected as acceptable. LLM-assisted reviewing was detected in 15.8% of ICLR reviews, boosting borderline papers by +4.9pp.
  • Dissemination: paper-to-poster/slides/video/social-media tools are emerging but suffer fidelity loss and need human layout fixes.

Five cross-cutting conclusions

1. AI is strong at structured, retrieval-driven tasks; weak at open-ended judgment (novelty assessment, experimental design, impact evaluation). 2. Generation outpaces verification: a paper costs $15 and hours; validating it takes days to weeks. 3. The most reliable mode is human-governed collaboration, not full autonomy — AI handles mechanical friction; humans keep judgment, interpretation, design, and accountability. 4. Effective systems converge on a layered architecture: exploration → tool execution → verification. Orchestration and provenance matter as much as model scale. 5. AI use is now a governance problem, not a detection problem: disclosure, attribution, responsibility, and research integrity — not AI-text detection.

End-to-end systems and open challenges

Four system paradigms are analyzed (sequential pipelines like The AI Scientist/FARS/ARIS; search & self-improving like Dolphin; skill/tool-integrated; multi-agent community simulators like ResearchTown). None has consistently met acceptance standards at major venues. Eight open challenges include cross-phase faithfulness, scientific judgment, reproducibility and accountability, citation provenance, governance, cross-domain generalization, cognitive ownership, and reliable AI-assisted research.

Bottom line

The core challenge is no longer whether AI can produce the forms of research, but whether it can preserve its substance: evidence, judgment, provenance, and accountability. For now, that remains an epistemological problem AI cannot solve — so keep humans in the loop.

Tags

#ai-research-automation#llm-agents#peer-review#research-integrity#human-ai-collaboration#survey#reproducibility#scientific-writing

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620509