AiScientist: Autonomous Long-Horizon Engineering for ML Research
> Paper: Toward Autonomous Long-Horizon Engineering for ML Research > arXiv: 2604.13018 | 2026-04-14 > Authors: Guoxin Chen, Jie Chen, Lei Chen, Jiale Zhao, Fanzhe Meng, Wayne Xin Zhao, Ruihua Song, Cheng Chen, Ji-Rong Wen, Kai Jia > Institutions: Gaoling School of AI, Renmin University of China; AweAI team > GitHub: https://github.com/AweAI-Team/AiScientist
Key points
- The core problem is long-horizon coherence, not single-step capability. An ML research project involves five tightly coupled, looping stages — task understanding, environment setup, code implementation, experiment iteration, and debugging. The paper frames the difficulty as a triple bind: *underspecification* (papers omit details), *delayed feedback* (results arrive slowly with confounded causes), and *system setup burden* (long dependency chains where a version difference changes everything).
- Dialogue-based multi-agent systems fail here. Frameworks like AutoGPT and MetaGPT pass context through conversation, but dialogue context is transient: windows overflow, handoffs lose information, and after many rounds agents forget why they made earlier decisions.
- Two design principles: (1) *Hierarchical orchestration* — a Tier-0 Orchestrator makes phase-level decisions like a PI; five Tier-1 specialist agents (Paper Comprehension, Prioritization, Implementation, Experimentation, Generic Helper) own defined roles and write scopes; bounded Tier-2 subagents handle focused subtasks. (2) *File-as-Bus* — agents coordinate by reading and writing durable files in a shared workspace instead of chatting. The result is "thin control over thick state": the orchestrator reads only concise summaries and a workspace map, while detailed plans, code, logs, and reports persist on disk.
- The workspace is a system of record. Plans (
plan.md), implementation and experiment logs, a submission registry, and final self-check reports are all durable artifacts. Runs are auditable, diffable, and resumable. - Two tracks, one architecture:
- *Paper track*: given a paper (markdown or zip), the system runs a full reproduction loop — comprehension, prioritization, implementation, experimentation, debugging, and a final self-check with
reproduce.sh. - *MLE track*: given a competition task, it iteratively improves a target metric. On a Detecting Insults task it ran autonomously for 23 hours, completing 74 experiment cycles with 18 best-so-far updates, raising validation AUC from 0.903 to 0.982.
- From dialogue systems to state systems: use the filesystem as shared memory, mirroring how human research teams actually collaborate via documents and logs rather than verbal handoffs.
- Thin control layers are stable: if the orchestrator had to remember code details, its context would bloat and decision quality would degrade.
- Permission scoping doubles as cognitive isolation: agents can only write to their own regions, preventing cross-role contamination of reasoning — enforced at the system level, not by prompting.
- Paper: arXiv:2604.13018 — Toward Autonomous Long-Horizon Engineering for ML Research
- Code: https://github.com/AweAI-Team/AiScientist
- PaperBench: Starace et al., 2025
- MLE-Bench: Chan et al., 2025
- Related systems: AutoGPT, MetaGPT, ChatDev, AIDE, RD-Agent
Evaluation results
PaperBench (48-hour budget):
| System | Average score | |--------|---------------| | Human PhD baseline | 41% | | AiScientist | 33.73 | | Strongest baseline | 22.58 | | AiScientist without File-as-Bus | 27.32 |
AiScientist improves over the strongest baseline by 10.54 points, narrowing the gap to the human PhD baseline from 19 to 7.27 points.
MLE-Bench Lite:
| System | Any Medal% | |--------|------------| | AiScientist | 81.82% | | Strongest matched baseline | 70.45% | | AiScientist without File-as-Bus | 50.00% |
Ablations: removing File-as-Bus costs -6.41 on PaperBench and -31.82 on MLE-Bench Lite Any Medal% — the steepest drop of any component. Hierarchical orchestration also contributes independently: even without File-as-Bus, the hierarchical variant beats BasicAgent by 4.74 points on PaperBench and by 22.73 points Above Median on MLE-Bench Lite. Simply adding more interaction (IterativeAgent) does not compensate for missing structure.
The deeper insight
> Long-horizon ML research engineering is fundamentally a systems problem, not a reasoning problem.
Stronger single-step capability or more interaction does not fix long-horizon failure if state decays over twenty iterations. File-as-Bus is not merely a communication protocol but a state governance mechanism: each agent re-grounds itself in the correct project state at every invocation. The sensitivity is highest on MLE-style tasks, where every iteration depends on accurate memory of what was tried, what worked, and why — information that becomes unrecognizable after 24 hours if it lives only in conversation context.
Design lessons for AI systems
Limitations
The authors note reliance on Docker sandboxing, limited evaluation coverage (only PaperBench and MLE-Bench Lite), substantial API cost for 24-hour runs, and a single-orchestrator point of failure. Future directions include broader long-horizon task types, human-in-the-loop checkpoints, cost optimization, and multi-orchestrator architectures.
Conclusion
AiScientist's three core contributions are the File-as-Bus protocol for durable state continuity, a hierarchical thin-control orchestration architecture, and the empirical insight that state continuity matters more than interaction volume. With 33.73 on PaperBench and 81.82% Any Medal on MLE-Bench Lite, it shows AI can sustain hours-to-days of autonomous research engineering under the right conditions — provided state management is engineered correctly at the system level.