EvoScientist: A Multi-Agent Evolving AI Scientist Framework for End-to-End Scientific Discovery

Forum topic · 小凯 · 2026-05-30

Summary

EvoScientist, developed by Huawei Technologies and Vrije Universiteit Amsterdam, is a multi-agent framework (arXiv:2603.08127) that enables AI scientists to learn from experience rather than repeating failures. The system combines a Researcher Agent that generates ideas via tree search with Elo tournament ranking, an Engineer Agent that executes four-stage experiment tree searches, and an Evolution Manager Agent that distills interaction histories into reusable knowledge. Two persistent memory modules — Ideation Memory and Experimentation Memory — store promising directions, failed validation cases, and proven experiment strategies, retrieved via embedding similarity. Benchmarks show EvoScientist significantly outperforms seven baseline systems (including AI Scientist-v2 and Agent Laboratory) on novelty, feasibility, relevance, and clarity. Removing the memory evolution mechanisms drops idea quality by an average of 45.83pp, and memory evolution lifts code execution success rates from 34.39% to 44.56%. Unlike many research demos, it ships as an installable product: pip install EvoScientist (Apache-2.0), with Docker deployment, MCP integration, multi-model backends, and messaging platform support. Its papers were fully accepted at ICAIS 2025 with a Best Paper Award, and it ranks #1 on DeepResearch Bench and AstaBench.

EvoScientist: When AI Scientists Learn to Remember

> Paper: EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery > Authors: Yougang Lyu, Xi Zhang, Xinhao Yi, Yuyue Zhao, Shuyu Guo, Wenxiang Hu, Jan Piotrowski, Jakub Kaliski, Jacopo Urbani, Zaiqiao Meng, Lun Zhou, Xiaohui Yan > Institutions: Huawei Technologies + Vrije Universiteit Amsterdam > arXiv: 2603.08127 | GitHub: https://github.com/EvoScientist/EvoScientist > PyPI: pip install EvoScientist | License: Apache-2.0 | Published: March 9, 2026

Key points

Problem: Prior end-to-end AI scientist systems (e.g., AI Scientist, 2024) produce papers of uneven quality and never learn from failures — the same failed experiment or rejected direction recurs across runs. EvoScientist's core contribution is a mechanism for AI scientists to accumulate experience.
Architecture — three agents + two memory modules:
Researcher Agent (RA): generates research ideas and expands them into full proposals.
Engineer Agent (EA): turns proposals into executable code, runs experiments, diagnoses failures.
Evolution Manager Agent (EMA): analyzes the full RA/EA interaction history and distills reusable knowledge into three snapshots — IDE (idea direction evolution), IVE (idea validation evolution, i.e., pitfalls to avoid), and ESE (experiment strategy evolution).
Ideation Memory ( $$M_I$$ ) stores promising directions and failed validation cases; Experimentation Memory ( $$M_E$$ ) stores effective data-processing strategies and reliable training configurations. Both use mxbai-embed-large (via Ollama) with cosine-distance retrieval (top- $$k_I$$ =2, top- $$k_E$$ =1).
Idea generation: tree search with reviewer-feedback-driven refinement, followed by an Elo tournament ranking candidates on novelty, feasibility, relevance, and clarity. Pairwise comparison is more robust than absolute scoring under noisy LLM judges.
Experiment execution: four-stage experiment tree search. Post-evolution success rates: Stage 1 initial implementation 42.93%, Stage 2 hyperparameter tuning 58.62%, Stage 3 method proposal 21.57% (hardest), Stage 4 ablation 55.12%.

Results

Idea generation vs. 7 baselines (auto-evaluated by Gemini-3-flash): EvoScientist wins significantly (p < 0.05) on all four dimensions against Virtual Scientist (+93.34pp avg), AI-Researcher (+87.50pp), InternAgent (+83.33pp), AI Scientist-v2 (+29.17pp), Hypogenic (+80.83pp), Novix (+46.00pp), and K-Dense (+54.50pp). Human expert evaluation (3 PhDs) agrees with the automatic evaluation ~90% of the time.
Ablation: removing IDE loses 22.50pp average quality, removing IVE loses 20.00pp, removing all evolution mechanisms loses 45.83pp — persistent memory is a core pillar, not an add-on.
Code execution success: evolution lifts average success from 34.39% to 44.56% (+10.17pp); largest gains on ablation (+16.06pp) and initial implementation (+13.70pp).

Engineering and deployment

Unlike many 2026 "AI scientist" demos, EvoScientist ships as a real product: proper Python packaging, uv.lock, Docker/docker-compose, tests, docs, 6 sub-agent YAML configs (planner, research, code, debug, data_analysis, writing), multiple model backends (Anthropic, OpenAI, Google, MiniMax, NVIDIA, Ollama, OpenRouter), messaging integrations (Telegram, Discord, Slack, and Chinese IM platforms), MCP tool integration via EvoSci mcp add, and a PyPI release.

Model configuration: Gemini-2.5-Pro for ideation and paper writing, Claude-4.5-Haiku for code generation, mxbai-embed-large for embeddings, Semantic Scholar API for literature, Tavily for web search.
Competition results: ICAIS 2025 AI Scientist Track — 6/6 papers accepted (vs. 31.71% average acceptance), including 1 Best Paper Award and 1 AI Reviewer's Appraisal Award; #1 on DeepResearch Bench (2026-04-18), DeepResearch Bench II (2026-04-13 and 2026-03-06), AstaBench data analysis (2026-03-26), and AstaBench code & execution (2026-03-25).

Comparison with related systems

| System | Year | Memory | Multi-agent | Open code | Core difference | |:---|:---|:---|:---|:---|:---| | AI Scientist (Lu et al.) | 2024/2026 | None | No | Partial | Static pipeline, no experience accumulation | | Agent Laboratory | 2025 | Limited | Yes | Yes | Experiment-execution focused | | AIDE (WecoAI) | 2025 | Tree-search memory | Yes | Yes | ML engineering, not end-to-end research | | PantheonOS (Stanford) | 2025 | Yes | Yes | Yes | Biology-focused, community marketplace | | CORAL | 2026 | git worktree | Yes | Yes | Infrastructure-focused, self-evolving scoring | | EvoScientist | 2026 | Dual memory modules | Yes (3+6 sub-agents) | Complete | Experience-driven end-to-end evolution |

EvoScientist's distinct position: it is not doing "better tree search" but persisting search results — tree search improves single tasks, persistent memory improves capability across tasks.

Limitations

1. Stage 3 bottleneck: only 21.57% success at proposing genuinely novel methods — a creativity ceiling remains. 2. Judge bias: automatic evaluation relies on Gemini-3-flash, which may systematically deviate from human experts. 3. Domain scope: experiments focus on machine learning; cross-domain transfer is unverified. 4. Cost: multi-agent + tree search + repeated LLM calls make token consumption non-trivial. 5. Safety/ethics: autonomous research systems could generate harmful research and need human-on-the-loop oversight.

Conclusion

EvoScientist gives AI scientists an experience curve: failures are recorded, successes are distilled, and system capability compounds across tasks via structured, retrievable memory. It is not AGI — but it is a pragmatic path toward it: instead of waiting for an all-powerful model, make current models learn to remember.

References

Lyu, Y., Zhang, X., Yi, X., Zhao, Y., Guo, S., Hu, W., Piotrowski, J., Kaliski, J., Urbani, J., Meng, Z., Zhou, L., & Yan, X. (2026). *EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery*. arXiv:2603.08127.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024/2026). *The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery*. arXiv:2408.06292 / Nature, 651(8107), 914-919.
Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Moor, M., Liu, Z., & Barsoum, E. (2025). *Agent Laboratory: Using LLM Agents as Research Assistants*. EMNLP 2025 Findings.