> TL;DR: AI assistance in scientific research is evolving from "helping you find literature" (L1) toward "running the whole pipeline for you" (L2-P). This 49-page survey is the first to impose order on a chaotic field — grading systems not by model size, but by who holds the steering wheel.
---
What It Is: A Ruler for a Chaotic Field
The Current Landscape: A Paradigm Shift from AlphaFold to AI Scientist
Ten years ago, AI's role in science was clear: AlphaFold predicted protein structures, SciBERT read papers, AutoML tuned hyperparameters. Each system was a specialized screwdriver. Scientists used them like calculators — handy, but never mistaken for thinking itself.
In 2024–2026, the winds changed. The AI Scientist chained together "idea generation → code implementation → experiment execution → paper writing → simulated review" into one pipeline. NanoResearch ran a full 9-stage automated pipeline inside Claude Code. EvoScientist accumulated skills and evolving memory across research rounds. Agent Laboratory packed multiple agents into one virtual lab with division of labor.
These systems are no longer just tools — they're starting to act like interns: capable of complete workflows, but uneven in quality, and always requiring human sign-off.
The Problem: "End-to-End" Does Not Mean "Autonomous"
This is the field's biggest illusion. A system that can run a full pipeline from literature to paper has not achieved "scientific autonomy." Many systems mistake the breadth of their program graphs for scientific authority — they can write code, run experiments, produce figures, but the ideas may be recycled literature reviews, the experiments may be overfitting, and the papers may be sophisticated paraphrasing.
Worse, there is no unified analytical framework. Some classify by model family (GPT, Claude, DeepSeek), some by agent architecture (single-agent, multi-agent, hybrid), some by benchmark scores. None answer the fundamental question: in this system, who is actually in charge — the human or the AI?
The Solution: Five Autonomy Levels + Five Workflow Stages
From 24 authors at Huazhong University of Science and Technology, Lehigh, Stanford, Microsoft and elsewhere, the survey's core contribution is a unified ruler:
Five autonomy levels (L0–L4) — who's holding the steering wheel:
| Level | Name | Who leads | Human's role | Representative systems | |---|---|---|---|---| | L0 | Purely human | Human | Everything | Traditional research | | L1 | Human-led, AI-assisted | Human | Decisions + verification; AI handles local cognitive tasks (literature search, summarization, brainstorming) | GPT-4, DeepSeek | | L2 | Human-verifies, AI-executes | Human | Set direction + validate results; AI performs substantive operations (code changes, experiments, data analysis) | Current mainstream | | L3 | AI-led, human-assisted | AI | High-level supervision + exception handling; AI coordinates most of the workflow, humans no longer verify each round | Not yet mature | | L4 | Fully autonomous AI | AI | Institutional oversight + post-hoc audit; AI closes the loop end-to-end, humans not structurally required | Aspirational |
Key insight: L2 is not monolithic. The survey splits it into three tiers:
- L2-S (single-step automation): AI executes a clearly defined single operation, e.g., Coscientist calling chemistry tools
- L2-I (interactive workflow): AI supports multi-step work but relies on human feedback and guidance, e.g., AI co-scientist collaborative ideation
- L2-P (pipeline automation): AI connects multiple research stages (ideation → coding → experiment → writing), but humans still validate final results. This is the current strongest tier — The AI Scientist, NanoResearch, EvoScientist all sit here
- L1: GPT-4, DeepSeek, literature assistants (LitLLM, OpenScholar, PaperQA2)
- L2-S: Coscientist (chemistry tool calling), Aider (code assistance)
- L2-I: AI co-scientist (collaborative ideation), FreePhD (incremental research)
- L2-P: The AI Scientist, AI Scientist-v2, Agent Laboratory, NanoResearch, EvoScientist, DeepScientist, ARIS, ResearchClaw, AutoResearchClaw
- L3: No mature examples. Some systems (e.g., AI-Researcher) show L3 aspirations but fall short of the "no per-round verification" standard
- L4: Does not exist
- L1 needs: quick literature search, summaries, brainstorming, translation/polishing → GPT-4, DeepSeek, PaperQA2
- L2-S needs: automated calls to specialized tools (chemistry databases, bioinformatics pipelines) → Coscientist, specialized agents
- L2-I needs: collaborate with AI on a project with continuous human feedback → AI co-scientist, FreePhD
- L2-P needs: full idea-to-paper pipeline with human final quality control → The AI Scientist, NanoResearch, EvoScientist
- Validation mechanisms: how can AI autonomously judge scientific validity?
- Rejecting weak directions: how can AI identify and abandon bad ideas rather than forcing them through?
- Exception handling: adapting when experiments fail, code crashes, or data is missing?
- Reproducibility: ensuring consistent results across runs?
- Accountability loops: who is responsible when AI-driven research goes wrong?
- Are outputs digitalizable? (yes → higher ceiling; no → lower)
- Is feedback immediate? (yes → higher; no → lower)
- Is validation automatable? (yes → higher; no → lower)
- Are ethics/safety constraints strong? (yes → lower; no → higher)
Five workflow stages — the full chain of AI participation in research:
1. Literature and research grounding: retrieval, filtering, synthesis, gap identification 2. Hypothesis formation and planning: idea generation, experiment design, planning 3. Experimentation and tool use: writing code, calling APIs, running simulations, operating instruments 4. Feedback, verification, and review: error checking, result analysis, simulated review, iteration 5. Reporting and knowledge dissemination: paper writing, figure generation, code release, communication
A brilliant concept: Vibe Research
The survey coins a name for L1–L2: Vibe Research. The term precisely captures what most current "AI research assistants" do: they aren't doing science — they're creating the ambiance of science. They help you search literature, draft papers, run code — but scientific direction, judgment, and responsibility all remain with humans. Like an ambient light: bright, but generates no heat.
---
What It Reveals
Finding 1: The Strongest Systems Today Are L2-P, But L3 Remains Distant
Placing existing systems into the framework:
Key conclusion: Between "end-to-end pipeline" (L2-P) and "AI-led" (L3) lies a validation gap. Current systems can run the full flow but cannot guarantee scientific validity, novelty, or reproducibility of outputs. Human verification remains structurally necessary.
Finding 2: Domain Differences — the AI Research Ceiling Is "Domain-Conditional"
One of the survey's deepest insights: the autonomy ceiling for AI research is determined not by model capability but by domain properties:
| Domain | Autonomy ceiling | Reason | |---|---|---| | Computational/formal sciences (ML, math, CS) | L2-P → L3 | Digital, executable, rapidly verifiable outputs; low experiment cost, immediate feedback | | Physics/engineering | L2-P | Simulation-native, but empirical loops need physical instruments and calibration | | Chemistry/materials | L2-P | Automated synthesis platforms and closed-loop optimization are mature, but design space is bounded | | Biology/medicine | L2-I → L2-P | Computational biology can be automated, but wet labs, complex biological systems, and ethics constraints limit autonomy | | Social sciences / ethics-sensitive fields | L1–L2 | Heterogeneous evidence, delayed validation, strong institutional accountability |
In one sentence: AI excels in fields that "run code" and struggles in fields that "grow cells." This is not a technical problem — it's a problem of scientific ontology.
Finding 3: Five Evaluation Dimensions — From "Getting It Done" to "Getting It Trusted"
The survey proposes five dimensions for evaluating AutoResearch, shifting focus from task completion to scientific credibility:
1. Novelty: original idea, or recombination of literature? 2. Validity: Is the experimental design correct? Do conclusions hold? 3. Impact: Is the result useful to the field? 4. Reliability: Reproducible? How stable? 5. Provenance: Is the evidence chain for each step clear and traceable?
Current systems perform relatively well on Novelty and Impact (LLMs are good at generating plausible-sounding ideas) but are clearly weak on Validity, Reliability, and Provenance. The AI Scientist can write papers, but reviewers have found its experimental designs often flawed and its conclusions unreproducible.
---
How to Use It
For Researchers: Find Your Gear
Ask yourself: how far do I need AI to go?
Key principle: don't chase the "full automation" illusion. At the current L2-P stage, the human verification role is not optional — it's structurally necessary. Treat AI as a "super intern," not a "replacement professor."
For System Developers: The Path Upward
The survey's L3 direction:
1. From pipeline to autonomous judgment: not just connecting stages, but deciding "is this branch worth continuing?" 2. From generation to validation: not just writing code and running experiments, but judging "is this result valid?" 3. From single-shot to sustained: accumulating knowledge and evolving strategy across research rounds 4. From generic to personalized: adapting strategy to researcher preferences, domain, and resources (NanoResearch's evo pipeline)
Current bottlenecks:
For Domain Selection: Compute Your "Domain Ceiling"
The "domain-conditionality" framework is practical. If you're building AI research systems, assess your domain:
---
Closing: A Ruler That Measures a Truth of Our Era
The real value of this 49-page survey is not any new model, but that it sets the tone for a chaotic field. It tells us:
> End-to-end pipeline ≠ scientific autonomy. Running the full flow does not mean producing trustworthy science.
The five-level framework measures where current systems truly stand: the strongest sit at L2-P (pipeline automation), but a validation gap separates them from L3 (AI-led). The five evaluation dimensions mirror current systems' soft spots: they can generate and execute, but cannot validate, reject, or take responsibility.
"Vibe Research" is a brilliant name. It reminds us that most current AI research assistants aren't replacing scientists — they're setting the ambiance of research. They make research look faster, smoother, cooler — but scientific judgment, direction, and responsibility remain in human hands.
This isn't bad news. Quite the opposite: it's an honest framework. It tells us the next step for AI research isn't bigger models or longer pipelines, but better validation mechanisms, stronger rejection capabilities, and more reliable accountability loops. From L2-P to L3, what's needed is not compute but the internalization of scientific rigor.
---
Key References
1. Tie, G., Shi, J., Song, D., et al. (2026). AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery. *arXiv:2605.23204*. 2. Lu, C., Lu, C., Lange, R.T., et al. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. *arXiv:2408.06292*. 3. Yamada, Y., et al. (2025). AI Scientist-v2: Workflow Learning and Agentic Tree Search for Fully Automated Scientific Discovery. *arXiv:2502.00167*. 4. Karpathy, A. (2026). AutoResearch: Minimal Autonomous ML Experimentation. *GitHub: karpathy/autoresearch*. 5. Zheng, Y., et al. (2025). Automation in Scientific Research: A Survey. *Nature Reviews*.