EvoScientist: When AI Scientists Learn to Remember — A Multi-Agent Evolving Framework for End-to-End Research Automation

> Paper: EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery > Authors: Yougang Lyu, Xi Zhang, Xinhao Yi, Yuyue Zhao, Shuyu Guo, Wenxiang Hu, Jan Piotrowski, Jakub Kaliski, Jacopo Urbani, Zaiqiao Meng, Lun Zhou, Xiaohui Yan > Institutions: Huawei Technologies + Vrije Universiteit Amsterdam > arXiv: 2603.08127 | GitHub: https://github.com/EvoScientist/EvoScientist > PyPI: pip install EvoScientist | License: Apache-2.0 > Published: March 9, 2026

1. The Core Problem: Why AI Scientists Keep Repeating Mistakes

In 2024, The AI Scientist produced a paper in three days on a single GPU. The result stunned academia — but problems quickly surfaced. Its paper quality was uneven, and more fatally, it could not learn from failure. An experiment that crashed today crashed again tomorrow on a different topic; a direction rejected by reviewers last week resurfaced this week in slightly different packaging. The system behaved like a junior researcher whose memory resets on every reboot.

EvoScientist targets exactly this pain point: how to give an AI scientist long-term memory?

The authors' core observation is simple: human research skill grows not from perfect single-shot execution but from accumulated experience — knowing which directions are promising, which pitfalls to avoid, and which coding patterns are reliable.

2. Architecture: Three Agents + Two Persistent Memory Modules

| Agent | Role | Analogy | |:---|:---|:---| | RA (Researcher Agent) | Generates research ideas, expands them into full proposals | Lab PI | | EA (Engineer Agent) | Turns proposals into executable code, runs experiments, diagnoses failures | Lab engineer | | EMA (Evolution Manager Agent) | Analyzes full RA/EA interaction histories, distills reusable knowledge | Lab veteran |

RA and EA each carry a persistent memory module (a "mistake notebook"):

• Ideation Memory (\(M_I\)): Records promising research directions and failed validation cases. RA consults it before generating each new idea.
• Experimentation Memory (\(M_E\)): Records effective data-processing strategies and reliable model training configurations. EA consults it before writing code.

After each task, EMA reviews the RA–EA chat logs and distills three knowledge snapshots:

1. IDE (Idea Direction Evolution): promising directions extracted from Top-3 ideas 2. IVE (Idea Validation Evolution): pitfalls learned from failed experiments 3. ESE (Experiment Strategy Evolution): best practices distilled from successful code

These snapshots are written into \(M_I\) and \(M_E\) and retrieved in the next round of tasks.

3. Technical Details: Tree Search + Elo Tournament + Four-Stage Experiments

3.1 Idea Generation: Idea Tree Search

1. Starting from a rough direction, RA generates multiple candidate ideas 2. Each candidate receives "reviewer feedback" 3. Feedback is used to refine sub-ideas and expand the search tree 4. Candidates are ranked by an Elo tournament — pairwise comparisons scored on novelty, feasibility, relevance, and clarity 5. The Top-1 idea is expanded into a full research proposal (background, methods, experiment plan)

The Elo tournament only needs relative comparisons ("which is better"), which is more robust than absolute scoring under noisy LLM judges.

3.2 Experiment Execution: Four-Stage Tree Search

| Stage | Goal | Success rate (with evolution) | |:---|:---|:---| | Stage 1 | Initial implementation: get baseline code running | 42.93% | | Stage 2 | Hyperparameter tuning: find optimal configuration | 58.62% | | Stage 3 | Propose method: implement the paper's core innovation | 21.57% | | Stage 4 | Ablation studies: verify each component's contribution | 55.12% |

Stage 3 is hardest — having an AI propose genuinely valuable methodological innovation succeeds only 21.57% of the time. Even so, experiment strategy evolution (ESE) adds +1.24pp there, showing that remembering past method patterns helps.

3.3 Memory Retrieval: Embedding Similarity

Both memory modules use mxbai-embed-large (locally deployed via Ollama) with cosine-distance retrieval:

• \(M_I\): top-\(k_I\) = 2 (direction memory needs precision, not volume)
• \(M_E\): top-\(k_E\) = 1 (experiment strategies are more specific; take the most relevant)

Retrieved content is injected directly into RA/EA prompts, forming retrieval-augmented generation.

4. Results: Outperforming Seven Baselines

4.1 Idea Generation (automatic evaluation, Gemini-3-flash judge)

| System | Type | Avg. lead over baseline | |:---|:---|:---| | Virtual Scientist | Open source | +93.34pp | | AI-Researcher | Open source | +87.50pp | | InternAgent | Open source | +83.33pp | | AI Scientist-v2 | Open source | +29.17pp | | Hypogenic | Commercial | +80.83pp | | Novix | Commercial | +46.00pp | | K-Dense | Commercial | +54.50pp |

All per-dimension wins (novelty, feasibility, relevance, clarity) are statistically significant (p < 0.05). Human expert evaluation (3 AI-domain PhDs) agreed with the automatic judge ~90% of the time.

4.2 Ablation: The Value of Evolution

| Removed component | Novelty loss | Feasibility loss | Avg. loss | |:---|:---|:---|:---| | −IDE (direction evolution) | −66.67% | −50.00% | −22.50pp | | −IVE (validation evolution) | −43.33% | −63.33% | −20.00pp | | −all (all removed) | −80.00% | −83.33% | −45.83pp |

Removing all evolution mechanisms collapses idea quality — persistent memory is a core pillar, not a nice-to-have.

4.3 Code Execution Success Rate

| Stage | Without evolution | With evolution | Gain | |:---|:---|:---|:---| | Initial implementation | 29.23% | 42.93% | +13.70pp | | Hyperparameter tuning | 48.94% | 58.62% | +9.68pp | | Propose method | 20.33% | 21.57% | +1.24pp | | Ablation studies | 39.06% | 55.12% | +16.06pp | | Average | 34.39% | 44.56% | +10.17pp |

5. Engineering: Not a Demo, a Product

The GitHub repository includes:

• pyproject.toml: a proper Python package (v0.1.1), Apache-2.0
• uv.lock: a 943KB dependency lock file
• Dockerfile + docker-compose.yml: one-command containerized deployment
• tests/ and docs/ directories
• 6 sub-agent YAML configs: planner, research, code, debug, data_analysis, writing
• Multi-model backends: Anthropic, OpenAI, Google, MiniMax, NVIDIA, Ollama, OpenRouter
• Multi-channel access: Telegram, Discord, Slack, Feishu, WeChat, QQ
• MCP integration: EvoSci mcp add connects external tools in one command
• PyPI release: pip install EvoScientist

5.1 Model Configuration

| Task | Model | Notes | |:---|:---|:---| | Idea generation | Gemini-2.5-Pro | Creative tasks | | Code generation | Claude-4.5-Haiku | Coding tasks | | Paper writing | Gemini-2.5-Pro | Writing tasks | | Embeddings | mxbai-embed-large (Ollama) | Local deployment | | Literature search | Semantic Scholar API | Academic search | | Web search | Tavily API | Optional |

5.2 Competition Results

• ICAIS 2025 AI Scientist Track: 6/6 papers accepted (100% vs. 31.71% average acceptance)
• 1 Best Paper Award: *Adaptive Evidential Meta-Learning with Hyper-Conditioned Priors for Calibrated ECG Personalisation*
• 1 AI Reviewer's Appraisal Award: *Hierarchical Change Signature Analysis: A Framework for Online Discrimination of Incipient Faults and Benign Drifts in Industrial Time Series*
• DeepResearch Bench: #1 (2026-04-18)
• DeepResearch Bench II: #1 (2026-04-13, 2026-03-06)
• AstaBench Data Analysis: #1 (2026-03-26)
• AstaBench Code & Execution: #1 (2026-03-25)

6. Comparison with Other AI Scientist Systems

| System | Year | Memory | Multi-agent | Open source | Key difference | |:---|:---|:---|:---|:---|:---| | AI Scientist (Lu et al.) | 2024/2026 | None | Single | Partial | Static pipeline, no experience accumulation | | Agent Laboratory | 2025 | Limited | Yes | Yes | Experiment execution focus, memory not central | | AIDE (WecoAI) | 2025 | Tree search | Yes | Yes | ML engineering focus, not end-to-end | | PantheonOS (Stanford) | 2025 | Yes | Yes | Yes | Biology focus, community-market driven | | CORAL | 2026 | git worktree | Yes | Yes | Infrastructure focus, self-evolving scoring | | EvoScientist | 2026 | Dual memory | 3+6 sub-agents | Full | Experience-driven end-to-end evolution |

EvoScientist's unique position: it is not building a better tree search but persisting search results. Tree search makes a single task efficient; persistent memory grows capability across tasks.

7. Limitations

1. Stage 3 bottleneck: success rate for genuinely novel method proposals is only 21.57% — the "creativity ceiling" persists 2. Judge bias: automatic evaluation relies on Gemini-3-flash, which may systematically deviate from human experts 3. Domain scope: experiments focus on machine learning; cross-domain transfer is unverified 4. Cost: multi-agent + tree search + repeated LLM calls means high token consumption per task 5. Safety and ethics: autonomous research systems could produce harmful research (e.g., bioweapon design) and require human-on-the-loop oversight

8. Conclusion: An Experience Curve for AI Scientists

EvoScientist's core contribution in one sentence: it gives AI scientists an experience curve. The gap between novice and senior researchers is not single-experiment speed but tacit knowledge — "this has been done before," "this hyperparameter combination crashes," "reviewers like this direction." EvoScientist makes such tacit knowledge explicit, structured, and retrievable via its ideation and experimentation memories. Every failure is recorded; every success is distilled; the system improves with use.

This is not AGI — but it is a pragmatic path toward it: not waiting for an omnipotent model, but teaching existing models to remember.

References

• Lyu, Y., Zhang, X., Yi, X., Zhao, Y., Guo, S., Hu, W., Piotrowski, J., Kaliski, J., Urbani, J., Meng, Z., Zhou, L., & Yan, X. (2026). *EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery*. arXiv:2603.08127.
• Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024/2026). *The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery*. arXiv:2408.06292 / Nature, 651(8107), 914-919.
• Schmidgall, S., et al. (2025). *Agent Laboratory: Using LLM Agents as Research Assistants*. EMNLP 2025 Findings.