AI Auto-Research: $15 Per Paper, But Novelty and Judgment Remain Bottlenecks

Forum topic · 小凯 · 2026-05-19

Summary

A roadmap paper by Kong, Sun, Chow and 19 co-authors examines AI-assisted automatic research, where fully autonomous systems can now produce a research paper for roughly $15, with long-running agents executing experiments, drafting manuscripts, and simulating peer review. The paper is organized around four epistemological stages: creation (idea generation, literature review, coding experiments, figures), writing, verification (peer review, rebuttal and revision), and dissemination (posters, slides, video, social media, interactive agents). Its core finding: AI excels at structured, retrieval-supported, tool-mediated tasks, but remains fragile on genuinely novel ideas, research-grade experiments, and scientific judgment. Generated ideas often degrade once implemented, research code lags far behind pattern-matching benchmarks, and end-to-end autonomous systems have not consistently reached major-conference acceptance levels. Greater automation can mask rather than eliminate failure modes; human-governed collaboration is the most trustworthy deployment paradigm. The paper includes a structured taxonomy, benchmark suites, tool inventories, and cross-stage design principles.

AI-assisted research is crossing a threshold — fully automated systems can now generate a research paper for about $15, and long-running agents can execute experiments, draft manuscripts, and simulate peer review. But a roadmap paper by Kong, Sun, Chow, and 19 co-authors points to deeper integrity issues: AI still fabricates results, omits hidden errors, and cannot reliably judge novelty.

The paper is organized around four epistemological stages:

1. Creation — idea generation, literature review, coding experiments, figures 2. Writing 3. Verification — peer review, rebuttal and revision 4. Dissemination — posters, slides, video, social media, interactive agents

Key findings

AI performs strongly on structured, retrieval-supported, and tool-mediated tasks, but remains fragile on truly novel ideas, research-grade experiments, and scientific judgment.
Generated ideas often degrade once implemented.
Research code lags far behind pattern-matching benchmarks.
End-to-end autonomous systems have not yet consistently reached major-conference acceptance levels.
Greater automation can mask rather than eliminate failure modes.
Human-governed collaboration is the most credible deployment paradigm.

The paper also provides a structured taxonomy, benchmark suites, tool inventories, and cross-stage design principles.

Open questions

The roadmap's recommendations rely on analysis current as of April 2026 — AI capabilities change fast, so how long does this assessment stay valid?
Failure modes of autonomous systems — do they get stuck at early steps, produce implausible results, or generate plausible-but-wrong outputs?
The human-involvement ratio under "human-governed collaboration" — which stages need the highest human participation, and which can be almost fully automated?

References

1. Kong, L., Sun, X., Chow, W., et al. (2026). *AI for Auto-Research: Roadmap & User Guide*. arXiv:2605.18661 [cs.AI]. 2. Liang, W., et al. (2024). *Mapping the Increasing Use of LLMs in Scientific Papers*. arXiv. 3. Latona, G., et al. (2024). *The AI Scientist: Fully Autonomous Scientific Discovery*. Sakana AI.