MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery

Forum topic · 小凯 · 2026-06-07

Summary

MLEvolve is a self-evolving framework that enables LLM-based agents to continuously improve on long-horizon machine learning engineering (MLE) tasks. The system introduces three core components: Progressive MCGS, a graph-based search structure with cross-branch information flow and entropy-based progressive scheduling; Retrospective Memory, a two-layer memory combining a cold-start domain knowledge base with a dynamic global memory using RAG-style retrieval; and adaptive coding modes that decouple strategic planning from tactical code execution to limit cumulative errors. On MLE-Bench, the framework reportedly achieves a 51.3% average medal rate, 89.7% valid submission rate, and 12.1% gold medal rate, roughly halving standard runtime compared to prior approaches. Ablation studies show Retrospective Memory contributes the largest performance gain (-18.6% when removed), followed by Progressive MCGS (-13.1%) and adaptive coding modes (-9.8%). MLEvolve also reportedly surpasses DeepMind's AlphaEvolve on several mathematical algorithm optimization benchmarks, suggesting general-purpose evolution frameworks can match specialized systems. The team emphasizes controllability and interpretability, with logged evolution steps and human-set objectives, and has open-sourced the code on GitHub.

MLEvolve: When AI Learns to Self-Evolve — A Philosophical Experiment on Machines Discovering Machines

> Paper: MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery > Authors: Shangheng Du, Xiangchao Yan, Jinxin Shi (InternScience) > arXiv: 2606.015xx (note: actual arXiv ID to be updated upon release) > Categories: cs.AI, cs.CL

Overview

MLEvolve tackles a recursive question: what happens when AI not only executes tasks, but designs the algorithms that design algorithms? The framework lets large language model (LLM) agents continuously self-evolve across long-horizon machine learning engineering (MLE) tasks — multi-stage workflows involving problem understanding, exploratory data analysis, baseline construction, iterative optimization, and final validation.

Unlike existing MLE agents (e.g., SWE-Agent, OpenHands), which act as one-shot tools that restart from zero on each new task, MLEVolve operates as a "lifelong learner": after every task, the system becomes smarter through a self-evolving loop that distills experience into reusable strategy-level knowledge.

Background: From AutoML to Autonomous MLE

Early AutoML (Auto-WEKA, Auto-sklearn): systematic but unimaginative search over pipelines — a "diligent intern."
Neural Architecture Search (NASNet, EfficientNet): smarter exploration via RL/evolution, but confined to single tasks and fixed search spaces.
MLEvolve: targets full engineering workflows, aiming for machines that define their own search spaces, accumulate knowledge, and improve their own search strategies — shifting AutoML from *automation* toward *autonomy*.

Three Core Innovations

1. Progressive MCGS (Multi-Branch Graph Search)

Traditional tree search keeps branches isolated: a discovery in branch A never informs branch B. MCGS adds graph-based reference edges so that cross-branch information flows — when one branch finds an effective feature-engineering method, other branches can reference it instead of re-experimenting.

An entropy-based progressive schedule mimics human researchers:

Early phase (high entropy): broad exploration of diverse strategies
Middle phase: narrowing based on preliminary results
Late phase (low entropy): deep optimization of the best strategies

2. Retrospective Memory

A two-layer memory system, like an apprentice's craft manual:

Cold-start domain knowledge: static, verified best practices extracted from high-quality open-source projects and research papers
Dynamic global memory: experience accumulated from actual task execution, retrieved via RAG-style mechanisms (task encoding → semantic similarity matching → experience injection into the agent's context)

3. Adaptive Coding Modes

Decoupling strategic planning from code execution — the "general and combat engineer" split:

Strategic mode: high-level planning, pseudocode, solution frameworks
Tactical mode: concrete coding, testing, debugging
Hybrid mode: automatic switching; when obstacles appear, escalate from tactical back to strategic replanning

This decoupling guards against cumulative errors in long-horizon tasks: strategic decisions get stricter validation, and local tactical mistakes cannot derail the overall direction.

Experimental Results

MLE-Bench Performance

| Metric | MLEvolve | Previous SOTA | Improvement | |--------|----------|---------------|-------------| | Average medal rate | 51.3% | ~35% | +46% | | Valid submission rate | 89.7% | ~60% | +50% | | Gold medal rate | 12.1% | ~5% | +140% | | Standard runtime | 12h | 24h | 2× efficiency |

A 51.3% medal rate means MLEvolve places in the top 10% on more than half of real Kaggle-derived competition tasks.

Cross-Domain: vs. AlphaEvolve

Despite being a general-purpose MLE system, MLEvolve reportedly exceeds DeepMind's AlphaEvolve (a math-algorithm specialist) on several mathematical optimization benchmarks, particularly in combinatorial optimization and number-theoretic algorithms — suggesting general evolution frameworks can match or surpass specialized ones by learning from broader experience.

Ablation Study

| Configuration | Medal rate | Drop | |---------------|-----------|------| | Full MLEvolve | 51.3% | — | | w/o Progressive MCGS | 38.2% | -13.1% | | w/o Retrospective Memory | 32.7% | -18.6% | | w/o adaptive coding modes | 41.5% | -9.8% | | Base agent only | 19.4% | -31.9% |

Retrospective Memory contributes the most — evidence that memory/experience accumulation is central to intelligence. The three components show a multiplicative, not merely additive, synergy.

Implications and Safety Considerations

AutoML redefined: machines become learners rather than tools; humans become supervisors rather than operators.
Scientific discovery: promising applications in drug discovery, materials science, and theory-driven mathematics.
Recursive self-improvement: current MLEvolve improves *strategies* (how to search), not its own *architecture*; its evolution is bounded within a human-defined framework.
Safety posture: all evolution steps are logged, humans can review and intervene, and objectives are set by humans and cannot be self-modified.
Open source: code is available at https://github.com/InternScience/MLEvolve for reproducibility, community contribution, and safety auditing.

Conclusion

MLEvolve is both a technical breakthrough and a metaphor about learning and growth. It captures the core mechanism of growing intelligence: extracting patterns from experience, using patterns to guide the future, and continuously improving in a loop. As Feynman put it: "The core of knowledge is not facts, but process."

References

1. Du, S., Yan, X., Shi, J., et al. "MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery." arXiv preprint arXiv:2606.015xx (2026). 2. Yang, C., et al. "SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS 2024. 3. Wang, X., et al. "OpenHands: An Open Platform for AI Software Developers as Generalist Agents." arXiv 2024. 4. Google DeepMind. "AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms." 2025. 5. Phan, H., et al. "MLE-Bench: Evaluating Machine Learning Agents on Machine Learning Engineering." ICLR 2025. 6. Silver, D., et al. "Mastering the game of Go with deep neural networks and tree search." Nature 529, 484-489 (2016).