DeepSeek core researcher Deli Chen (a core contributor to DeepSeek-V1/V2/V3/V4/R1/Coder/MoE architectures) has published a 46-page survey, From Copilots to Colleagues: A Survey of Autonomous Research Agents, that is 99% written by his own agent framework, DeliAutoResearch. The paper is available at https://victorchen96.github.io/auto_research_survey.pdf
The production stats themselves are evidence of the paper's thesis: a first draft in 76 minutes, 6 days of iteration across 6 rounds and 108 interactions, 648,000 tokens consumed, producing 46 pages, 103 references, 7 figures, and 4 tables—with under 2 hours of the author's own time for work that previously took at least a month.
The L1-L5 Autonomy Taxonomy
The paper classifies research agents into five levels, modeled on SAE self-driving grades:
| Level | Name | Human Role | Autonomous Duration | Representative Systems | |-------|------|------------|---------------------|------------------------| | L1 | Auto-completion | Full control | None | GitHub Copilot | | L2 | Task execution (approval) | Approve each step | Minutes | ChatGPT + Code Interpreter | | L3 | Multi-step (checkpoints) | Checkpoint review | 10 min - 1 hour | Claude Code, AutoGPT | | L4 | Fully autonomous (bounded domain) | Evaluate final output only | Hours - days | Devin, SWE-Agent, AI Scientist, Claude Code auto mode | | L5 | Self-directed | Set domain only | Weeks - months | No complete system (Co-Scientist closest) |
Two key judgments: the current frontier is at L4, whose critical prefix is "bounded domain"—domains with clear verifiers (tests passing, experiments succeeding, theorems proving). The barrier to L5 is not model capability but evaluation capability: agents cannot yet reliably judge novelty and importance of research directions.
Four Architecture Patterns
- Single-agent loops: ReAct-style observe-reason-act-feedback iteration. Reflexion raised HumanEval from 67% to 91%; LATS with MCTS reached 94.4% at 5-20x cost. The bottleneck is context saturation—L3 systems typically degrade after 10-15 steps.
- Multi-agent systems: CAMEL (role-playing), AutoGen (flexible dialogue topologies), MetaGPT (encoding software-company SOPs, achieving 100% task completion vs. 67% for free-form ChatDev chat). Structured, document-mediated communication is key.
- Hierarchical orchestration: supervisor-worker patterns; Claude Code isolates context per sub-agent; graph-based orchestration can learn optimal collaboration topologies.
- Tool-augmented execution: SWE-Agent's Agent-Computer Interface design lifted SWE-bench from 3.8% to 12.5%; CodeAct represents all actions as executable Python code.
- Code agents are the most mature track due to clear verifiers and standardized benchmarks: Claude Code at 72% on SWE-bench Verified, OpenHands 41-53%, Devin 13.86%, SWE-Agent 12.5%.
- General agents are least reliable: AutoGPT collapses after 10-15 steps; Agentless—a two-step locate-then-patch pipeline, not an agent at all—reached 27% on SWE-bench Lite at far lower cost, raising the question of whether agent architectures are sometimes over-engineering.
- Science agents are limited by physical interfaces: Coscientist synthesized aspirin and fluorescent dyes, but feedback loops break in biology/materials science where cycles take weeks or months.
- AI Scientist's $15/paper pipeline covers ideation through automated peer review, but cannot judge what problems are worth studying—typical L4 limitation: it can execute, not choose.
- From Copilots to Colleagues: A Survey of Autonomous Research Agents, Deli Chen, https://victorchen96.github.io/auto_research_survey.pdf
Findings from 17 Systems
Six Open Problems
1. Cognitive loops: agents get stuck repeating failed patterns; metacognitive monitoring is immature. 2. Context limitations: effective (not nominal) context length degrades over long-horizon tasks. 3. Novelty evaluation: the core L5 blocker—no automated way to assess importance in open domains. 4. Reproducibility: nondeterministic agent outputs conflict with scientific norms. 5. Safety: dual-use risks in chemistry/biology; humans cannot supervise every step of hours-long autonomous runs. 6. Cost: L4 runs cost $5-$50 each; weeks-to-months L5 runs would explode compute costs.
The Meta-Narrative
The survey's own production matches its defined L4 level: end-to-end autonomous execution with human evaluation of the final output—but not L5, because the topic was human-given. Chen's own observation: "Code Agents are causing crazy inflation in computer science papers." When production cost drops from person-months to 2 hours of human oversight plus 6 days of machine time, output volume inflates and quality variance widens—so human work shifts from writing to selecting and verifying.
The title's arc says it all: L1-L2 agents are copilots (human-led), L3-L4 are interns (bounded independent tasks, human acceptance), L5 would be colleagues—agents that choose what to research and judge the results. SWE-bench resolution jumped from ~5% to 70%+ between 2024 and 2025, an L4 milestone, but L5 requires what no benchmark measures: research taste.
> "With CodeAgent, I can finally pick up many things I had shelved due to lack of energy." — Deli Chen
References