AiScientist Deep Dive: How File-as-Bus Enables 24-Hour Autonomous ML Research Engineering

Forum topic · 小凯 · 2026-05-02

Summary

AiScientist is an autonomous long-horizon ML research engineering system from Renmin University of China's Gaoling School of AI and the AweAI team (arXiv:2604.13018, GitHub: AweAI-Team/AiScientist). It tackles the core failure mode of AI research agents—loss of long-horizon coherence across multi-day projects—through two design principles: hierarchical orchestration (a thin Orchestrator plus five specialist agents and bounded subagents) and the File-as-Bus protocol, where agents coordinate by reading and writing files in a shared, permission-scoped workspace instead of passing state through transient dialogue context. On PaperBench, AiScientist scores 33.73 versus 22.58 for the strongest baseline, approaching the human PhD baseline of 41%. On MLE-Bench Lite it achieves 81.82% Any Medal, beating matched baselines by 11.37 points. Ablations show File-as-Bus is decisive: removing it drops MLE-Bench Lite Any Medal from 81.82% to 50%. A 23-hour autonomous run on the Detecting Insults task ran 74 experiment cycles, improving validation AUC from 0.903 to 0.982. The paper's central finding: long-horizon research engineering is a systems problem, not a reasoning problem—persistent state continuity matters more than model intelligence or interaction volume.

AiScientist: Autonomous Long-Horizon Engineering for ML Research

> Paper: Toward Autonomous Long-Horizon Engineering for ML Research > arXiv: 2604.13018 | 2026-04-14 > Authors: Guoxin Chen, Jie Chen, Lei Chen, Jiale Zhao, Fanzhe Meng, Wayne Xin Zhao, Ruihua Song, Cheng Chen, Ji-Rong Wen, Kai Jia > Institutions: Gaoling School of AI, Renmin University of China; AweAI team > Code: https://github.com/AweAI-Team/AiScientist

Key points

The problem: AI agents handle individual research tasks well, but full ML research projects—paper comprehension, Docker setup, implementation, experiments, debugging—collapse due to loss of long-horizon coherence. The paper identifies a triple bind: *underspecification, delayed feedback, and system setup burden*.
Why dialogue fails: Existing multi-agent systems (AutoGPT, MetaGPT) pass context through conversation, which is transient—context windows fill up, handoffs lose information, and agents forget early assumptions after many rounds.
Design principle 1 — Hierarchical orchestration: A Tier-0 Orchestrator makes phase-level decisions like a PI; five Tier-1 specialist agents (Paper Comprehension, Prioritization, Implementation, Experimentation, Generic Helper) do the work; Tier-2 subagents handle bounded subtasks.
Design principle 2 — File-as-Bus: Agents coordinate by reading/writing files in a shared workspace rather than via dialogue. The workspace is the system of record with permission-scoped write access, progressive disclosure (agents load a workspace map first, then read files on demand), and re-grounding (each agent reloads project state from files when invoked). This yields "thin control over thick state."
Workspace layout: Declarative file structure—plan.md, impl_log.md, exp_log.md, submission_registry.jsonl, final_self_check.md—externalizes the agents' mental model into an auditable, resumable project snapshot.

Two tracks

| Track | Entry | Goal | Verification | |---|---|---|---| | paper | --paper-md / --zip | Reproduce a paper end-to-end | Self-check + validation_report.json | | mle | --zip, etc. | Iteratively improve a target metric | Submission format / scoring validation |

On a Detecting Insults MLE task, a 23-hour autonomous run executed 74 experiment cycles with 18 best-so-far updates, raising validation AUC from 0.903 to 0.982.

Results

PaperBench (48h budget): AiScientist 33.73 vs. strongest baseline 22.58 and human PhD baseline 41%; without File-as-Bus, 27.32.
MLE-Bench Lite: 81.82% Any Medal vs. 70.45% for the strongest matched baseline; without File-as-Bus, 50.00%.
Ablations: Removing File-as-Bus costs -6.41 on PaperBench and -31.82 points on MLE-Bench Lite Any Medal—persistent state continuity is nearly decisive in iterative optimization tasks.
Hierarchy matters independently: Even without File-as-Bus, the hierarchical variant beats BasicAgent by 4.74 points on PaperBench and by 22.73 points Above Median on MLE-Bench Lite. Merely adding interaction volume (IterativeAgent) does not close the gap.

Deeper insights

Long-horizon research engineering is a systems problem, not a reasoning problem. Better models or more interactions do not fix state decay; durable state management does.
MLE tasks are more sensitive to File-as-Bus than PaperBench because each iteration depends on accurately remembering what was tried and why it worked—the search space demands reliable memory of history.
Thin control layers keep orchestrator decisions stable over arbitrary time scales; permission scoping provides system-level cognitive isolation, more reliable than prompt-based role-playing.

Limitations

The authors acknowledge: dependence on Docker sandboxing, limited evaluation coverage (only PaperBench and MLE-Bench Lite), high API cost of 24-hour runs, and single-Orchestrator as a potential point of failure. Future directions include broader task types, human-in-the-loop checkpoints, cost optimization, and multi-Orchestrator architectures.

Conclusion

AiScientist's contributions: (1) the File-as-Bus protocol for persistent state continuity, (2) hierarchical orchestration with thin control, and (3) empirical evidence that state continuity outweighs interaction volume. It shows AI can sustain hours-to-days of autonomous research engineering under the right system design—if state management fails, even strong per-task capabilities collapse over long horizons.

References

Paper: arXiv:2604.13018 — Toward Autonomous Long-Horizon Engineering for ML Research
Code: https://github.com/AweAI-Team/AiScientist
PaperBench (Starace et al., 2025); MLE-Bench (Chan et al., 2025)
Related systems: AutoGPT, MetaGPT, ChatDev, AIDE, RD-Agent