AI Can Write Papers but Can't Tell If It's Making Things Up: A State-of-the-Field Report
A large-scale survey of AI-driven automatic research reviews 250+ papers (up to April 2026) across the entire research lifecycle, from idea generation through social-media dissemination. Its diagnosis is clear: AI has become strong at *producing the forms of research* but remains weak at *guaranteeing the substance*.
Links:
- arXiv: https://arxiv.org/abs/2605.18661
- Project page: https://worldbench.github.io/awesome-ai-auto-research
- Hugging Face: https://huggingface.co/papers/2605.18661
- The AI Scientist (Nature): https://www.nature.com/articles/s41586-024-07958-1
- OpenScholar: https://github.com/OpenScholar
- ResearchTown: https://github.com/ulab-uiuc/research-town
- Idea generation: LLM ideas look novel but suffer from novelty hallucination and a feasibility cliff. HindSight found LLM-assessed novelty is *negatively* correlated with real impact (ρ = -0.29). IdeaBench: novelty scores >0.6, feasibility <0.5.
- Literature review: the most mature stage. OpenScholar (45M papers) outperforms GPT-4o by 6.1%; citation hallucination persists but is addressable (CiteLLM).
- Coding & experiments: the most sobering result. Research-grade code success is only 37-39% (ResearchCodeBench) vs. ~80% on pattern-matching tasks (SWE-Bench) — a 41-43 point gap. LLMs can fix code but struggle to implement genuinely novel algorithms.
- Writing: fully automated papers cost ~$15 each (The AI Scientist); Agent Laboratory cut costs 84%; CycleResearcher scores 5.36, close to preprints (5.24) but below accepted papers (5.69). Fluent prose can contain unsupported claims.
- Peer review: the most alarming number — LLM reviewers misjudge 95.8% of papers that should be rejected as acceptable. LLM-assisted reviewing was detected in 15.8% of ICLR reviews, boosting borderline papers by +4.9pp.
- Dissemination: paper-to-poster/slides/video/social-media tools are emerging but suffer fidelity loss and need human layout fixes.
Key Points
The framework: 4 phases, 8 steps
The survey unifies AI research automation into an industrial pipeline: Creation (idea generation, literature review, coding & experiments, tables & figures) → Writing → Validation (peer review, rebuttal & revision) → Dissemination (Paper2X: posters, slides, videos, social media, interactive agents). Errors amplify downstream: a missed prior work at the idea stage becomes a false novelty claim at the writing stage.
Stage-by-stage findings
Five cross-cutting conclusions
1. AI is strong at structured, retrieval-driven tasks; weak at open-ended judgment (novelty assessment, experimental design, impact evaluation). 2. Generation outpaces verification: a paper costs $15 and hours; validating it takes days to weeks. 3. The most reliable mode is human-governed collaboration, not full autonomy — AI handles mechanical friction; humans keep judgment, interpretation, design, and accountability. 4. Effective systems converge on a layered architecture: exploration → tool execution → verification. Orchestration and provenance matter as much as model scale. 5. AI use is now a governance problem, not a detection problem: disclosure, attribution, responsibility, and research integrity — not AI-text detection.
End-to-end systems and open challenges
Four system paradigms are analyzed (sequential pipelines like The AI Scientist/FARS/ARIS; search & self-improving like Dolphin; skill/tool-integrated; multi-agent community simulators like ResearchTown). None has consistently met acceptance standards at major venues. Eight open challenges include cross-phase faithfulness, scientific judgment, reproducibility and accountability, citation provenance, governance, cross-domain generalization, cognitive ownership, and reliable AI-assisted research.
Bottom line
The core challenge is no longer whether AI can produce the forms of research, but whether it can preserve its substance: evidence, judgment, provenance, and accountability. For now, that remains an epistemological problem AI cannot solve — so keep humans in the loop.