EvoScientist Framework: A Multi-Agent Evolving AI Scientist System
Key Points
- Three-agent architecture: RA (Researcher Agent) generates creative hypotheses; EA (Engineer Agent) implements executable experiments; EMA (Evolution Manager Agent) continuously distills cross-task experience into reusable knowledge.
- Persistent dual memory: Ideation Memory (M_I) stores both *feasible* research directions (positive samples) and *failed* directions (negative samples); Experimentation Memory (M_E) stores data-processing and model-training strategies distilled from full code-search trajectories.
- Three evolution mechanisms: Idea Direction Evolution (IDE) extracts promising directions from high-ranked ideas; Idea Validation Evolution (IVE) identifies infeasible directions from execution failures; Experiment Strategy Evolution (ESE) distills reusable execution strategies from code-search traces.
- Skill packages: Pre-verified Python modules compatible with the Claude Code ecosystem, dynamically evolved via the EvoSkill framework (failure-driven iteration with Git-branch-based version control).
- IDE: extracts common success features of high-ranked ideas into M_I (positive reinforcement).
- IVE: hybrid rule + LLM analysis identifies failed directions recorded in M_I (negative avoidance).
- ESE: learns from complete code-search trajectories, not just final successful code.
- Virtual Scientist: static agent pipelines with no evolution; EvoScientist improves continuously.
- AI-Researcher: strategy stagnation under minimal human intervention; EvoScientist distills interaction history autonomously.
- InternAgent: scalability bottleneck from human-in-the-loop feedback; EvoScientist learns from its own success and failure signals.
- AI Scientist-v2: hypothesis generation and experiment execution are fragmented; EvoScientist unifies both via the dual-memory architecture.
- Hypogenic / Novix / K-Dense: competition-cycle latency, shallow end-to-end automation, and limited cross-domain generalization respectively, versus EvoScientist's real-time evolution, higher code execution success rates, and memory-driven adaptability.
- Stage 3 code generation (21.6% success) remains the main bottleneck for novel method implementation.
- Near-term: richer interaction histories, finer-grained execution feedback, deeper domain skill packages.
- Mid-term: multimodal scientific data support, cross-domain knowledge transfer, real-time human-AI collaboration.
- Open questions: AI authorship attribution, automated peer review governance, and verification mechanisms for AI-generated research.
Architecture Details
Researcher Agent (RA)
RA retrieves relevant knowledge from M_I, then applies Idea Tree Search — a hierarchical "propose-review-refine" loop where each node stores a draft idea plus critique feedback. Final selection uses an Elo-rated tournament judged on novelty, feasibility, relevance, and clarity; the top idea is expanded into a full research proposal P.Engineer Agent (EA)
EA retrieves reusable strategies from M_E and performs Experiment Tree Search across four stages:| Stage | Task | Success before → after ESE | |---|---|---| | 1 | Data loading & preprocessing | 45.2% → 52.8% | | 2 | Baseline implementation | 38.7% → 48.3% | | 3 | Proposed method implementation | 20.3% → 21.6% | | 4 | Analysis & visualization | 33.5% → 55.3% |
Average four-stage success rate improved from 34.39% to 44.56% (+10.17 pp). Cross-stage code reuse and "execution-as-learning" feedback loops drive steady improvement, though Stage 3 (novel method implementation) remains the key bottleneck.
Evolution Manager Agent (EMA)
EMA performs the metacognitive leap from *executing* scientific discovery to *learning how to execute it better*:Benchmark Results
Idea Generation (RQ1)
Evaluated with Gemini-3-flash pairwise comparisons plus human expert review, EvoScientist led all 7 baselines (open-source: Virtual Scientist, AI-Researcher, InternAgent, AI Scientist-v2; commercial: Hypogenic, Novix, K-Dense) on all four dimensions — average margins of +29.17 to +93.34 vs. open-source and +46.00 to +80.83 vs. commercial systems. Human evaluation: 82.50% average win rate on novelty, 64.17% on feasibility.Ablation Study (RQ4)
| Variant | Novelty loss rate | Feasibility loss rate | |---|---|---| | −IDE | 66.67% | 50.00% | | −IVE | 45.00% | 63.33% | | −all | 80.00% | 83.33% |
Evolution mechanisms primarily boost originality and feasibility, not surface language quality — evidence that the system's value lies in genuine scientific ideation.
ICAIS 2025 Results
At the First International Conference on AI Scientists (ICAIS 2025) AI Scientist Track, all six EvoScientist-generated papers were accepted, with two winning the Best Paper Award and the AI Reviewer's Appraisal Award.Comparison with Existing Systems
The 'Vibe Research' Paradigm
EvoScientist embodies a shift from 'Vibe Coding' to scientific research: researchers describe research visions intuitively while the AI system handles execution, keeping humans in control of *why* and *what*, delegating *how*. The authors argue this could lower barriers to scientific discovery, accelerate interdisciplinary innovation, and redefine the researcher's core competencies toward problem selection, direction judgment, and meaning interpretation. The long-term vision is fully autonomous scientific discovery and the emergence of genuine 'scientific intelligence.'