MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery

Forum topic · 小凯 · 2026-06-07

Summary

MLEvolve is a self-evolving framework that enables large language model (LLM) agents to continuously improve on long-horizon machine learning engineering (MLE) tasks. The framework combines three core components: Progressive MCGS, a graph-based search method that enables cross-branch information sharing with entropy-based progressive scheduling; a Retrospective Memory system with a cold-start domain knowledge base and a dynamic global memory using retrieval-augmented generation; and adaptive coding modes that decouple strategic planning from tactical code execution to reduce cumulative errors. On MLE-Bench, built from real Kaggle competitions, MLEvolve reportedly achieves a 51.3% average medal rate and 89.7% valid submission rate, roughly halving standard runtime from 24 to 12 hours. Ablation studies show Retrospective Memory contributes the largest performance gain, followed by Progressive MCGS and adaptive coding modes. The framework also reportedly outperforms DeepMind's AlphaEvolve on several mathematical optimization benchmarks, suggesting general evolutionary mechanisms can match specialized systems. The authors position MLEvolve as a shift from automated to autonomous machine learning, emphasizing controllability and interpretability, with open-source code released to the community.

Overview

Paper: MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery Authors: Shangheng Du, Xiangchao Yan, Jinxin Shi (InternScience) arXiv: 2606.015xx (placeholder ID, to be updated upon release) Categories: cs.AI, cs.CL Posted: 2026-06-07

This post introduces MLEvolve, a framework that lets LLM agents continuously self-evolve on long-horizon machine learning engineering (MLE) tasks. The central question: what happens when AI starts designing the algorithms that design algorithms?

Background: From AutoML to Long-Horizon MLE

Early AutoML systems (Auto-WEKA, Auto-sklearn) automated feature engineering, hyperparameter tuning, and model selection through exhaustive search — effective but rigid.
Neural architecture search (NASNet, EfficientNet) introduced RL and evolutionary methods, but remained confined to single tasks and fixed search spaces.
Long-horizon MLE tasks go further: understanding vague requirements, exploratory data analysis, building baselines, iterative optimization, and validated delivery — like supervising a graduate student through a months-long research project.
Existing MLE agents (SWE-Agent, OpenHands) lack continuous learning: every new task starts from scratch.

Three Core Innovations

1. Progressive MCGS (Progressive Multi-Branch Graph Search)

Instead of a tree where branches are isolated, MCGS adds graph-based reference edges so different branches share discoveries (e.g., a working feature-engineering strategy found in branch A can be reused by branch B). An entropy-based progressive schedule drives broad exploration early, narrowing mid-run, and deep optimization of the best strategy late.

2. Retrospective Memory

A two-layer memory system:

Cold-start domain knowledge: static, verified best practices distilled from open-source projects and papers.
Dynamic global memory: experience accumulated from actual task execution, retrieved via RAG-style semantic similarity matching and injected as prior knowledge into new tasks.

3. Adaptive Coding Modes

Planning and execution are decoupled, like a general and combat engineers:

Strategic mode: high-level planning, solution frameworks, pseudo-code.
Tactical mode: concrete coding, testing, debugging.
Hybrid mode: automatic switching when obstacles require re-planning.

This decoupling guards against cumulative errors in long-horizon runs: major strategic decisions get stricter validation, and local tactical mistakes cannot derail the overall direction.

Experimental Results

MLE-Bench (real Kaggle-derived competitions)

| Metric | MLEvolve | Prior SOTA | Gain | |--------|----------|------------|------| | Average medal rate | 51.3% | ~35% | +46% | | Valid submission rate | 89.7% | ~60% | +50% | | Gold medal rate | 12.1% | ~5% | +140% | | Standard runtime | 12h | 24h | 2x efficiency |

Comparison with AlphaEvolve

Despite being a generalist framework, MLEvolve reportedly outperforms DeepMind's AlphaEvolve (a math-specialized system) on several mathematical optimization benchmarks, notably combinatorial optimization and number-theoretic algorithms — evidence that general evolutionary mechanisms can match specialized ones by learning from broader experience.

Ablation Study

| Configuration | Medal rate | Drop | |---------------|-----------|------| | Full MLEvolve | 51.3% | – | | – Progressive MCGS | 38.2% | -13.1% | | – Retrospective Memory | 32.7% | -18.6% | | – Adaptive coding modes | 41.5% | -9.8% | | Base agent only | 19.4% | -31.9% |

Retrospective Memory contributes the most, and the three components show multiplicative synergy rather than additive gains.

Implications and Outlook

Paradigm shift: from automation (human-defined search spaces, machine as tool) toward autonomy (machine defines, searches, accumulates knowledge, and improves its own strategy; human as supervisor).
Scientific discovery: potential applications in drug discovery, materials science, and theoretical physics.
Recursive self-improvement boundaries: current MLEvolve improves its *strategies*, not its own architecture or goals. The team emphasizes controllability and interpretability — logged evolution steps, human oversight, and human-set objectives.
Open source: code at https://github.com/InternScience/MLEvolve

Conclusion

The post closes philosophically: MLEvolve touches the core mechanism of growth — extracting patterns from experience, using patterns to guide the future, and improving continuously in a loop. Perhaps true intelligence is not a moment of insight, but the capacity for continuous evolution itself.