AutoScientists: Self-Organizing AI Agent Teams for Long-Running Scientific Research (Harvard)

Forum topic · 小凯 · 2026-06-03

Summary

AutoScientists, a system from Shanghua Gao, Ada Fang, and Marinka Zitnik at Harvard, replaces single-agent and centrally coordinated multi-agent approaches with a decentralized, self-organizing team of LLM agents for long-running scientific experimentation. Agents coordinate through shared state rather than a central orchestrator, alternating Discussion phases (analyzing the champion model, critiquing proposals, and self-organizing into teams) with Execution phases (parallel propose-execute loops). Key mechanisms include pre-execution proposal critique, cross-team dead-end registries that turn individual failures into shared negative knowledge, and consensus-driven team reorganization. Default configuration uses 3 Analyst agents and 6 Experiment agents. Benchmarks: 74.40% average leaderboard percentile on BioML-Bench (24 biomedical ML tasks, +8.33% over the strongest baseline), 1.9x experiment efficiency on GPT NanoChat training optimization (val_bpb ~0.978 with 34 vs 65 experiments, later reaching 0.9730), and improvement of protein fitness prediction from Kermut SOTA (Spearman rho 0.747 to 0.840 on ACE2-Spike, with frozen recipes yielding +6.5% across 217 ProteinGym assays). Ablations show complementary contributions from analysts, cross-agent feedback, self-organization, and shared state. Paper: arXiv:2605.28655; code on GitHub (mims-harvard/AutoScientists).

Overview

Paper: AUTOSCIENTISTS: Self-Organizing Agent Teams for Long-Running Scientific Experimentation Authors: Shanghua Gao*, Ada Fang*, Marinka Zitnik (Harvard University) Paper: https://arxiv.org/abs/2605.28655 Code: https://github.com/mims-harvard/AutoScientists Project page: https://autoscientists.openscientist.ai

Why single agents are not enough

Scientific research is a long-running loop — hypothesize, design, run experiments, revise, repeat. It requires parallel exploration of competing hypotheses, dynamic re-routing as evidence accumulates, and memory of past failures.

Limitations of existing approaches:

| Approach | Limitation | |------|--------| | Single agent (AIDE, Autoresearch) | Single-trajectory search, no parallel exploration | | Multi-agent with coordinator (PI-Scientist-Critic) | Central orchestrator fixes the search decomposition up front | | Debate frameworks | Converge toward consensus instead of filtering weak proposals |

Core insight: long-running experimentation needs a decentralized, self-organizing team — agents decide research directions, form teams, critique each other's proposals, and share failures on their own.

Architecture: self-organizing agent teams

Decentralized, no central orchestrator

All agents coordinate through shared state: a champion model p*, an experiment log L, a shared forum F of structured discussion posts, and team-local state (queues, dead-end registries, hypotheses). Everything, including failures, is visible to all agents — one agent's pitfall is instantly known team-wide.

Two-phase cycle

| Phase | Activity | |------|--------| | Discussion | Agents analyze the champion model, propose directions, critique proposals, identify gaps in the search space, and self-organize into K teams | | Execution | Teams run parallel propose-execute loops until stalled |

Ten consecutive non-improving experiments by a team triggers a new Discussion phase and possible reorganization.

Agent roles

1. Analyst agents — maintain the team's search knowledge, audit the experiment log for untested directions, rank the proposal queue by observed effect size, and propose variants when the champion updates. 2. Experiment agents — claim experiments from the team queue, apply code changes to p*, train, log results, and double-confirm improvements within the noise band.

Default configuration: 3 Analysts + 6 Experiment agents.

Benchmark results

BioML-Bench (24 biomedical ML tasks)

| System | Avg. leaderboard percentile | |------|---------------------------| | AutoScientists | 74.40% (+8.33% vs strongest baseline) | | Autoresearch | 66.07% | | Biomni | lower |

Largest gain in drug discovery: 46.16% → 64.52%. Protein engineering saturated at 96.97% for both. All 24 tasks completed.

GPT NanoChat training optimization

From scratch: AutoScientists reached val_bpb ≈ 0.978 in 34 experiments vs 65 for Autoresearch — 1.9x more efficient.
From the champion: AutoScientists found 7 improvements in 93 experiments → 0.9730; Autoresearch accepted 0 of 100.
AutoScientists' first improvement (query-key normalization order) was never proposed in Autoresearch's 100 attempts — evidence of a wider hypothesis space, not more compute.

ProteinGym fitness prediction

Starting from SOTA Kermut (ACE2-Spike binding: Spearman ρ=0.747), AutoScientists discovered a three-GP ensemble, extended zero-shot features, greedy diversity feature selection, and a quantile-warping objective, reaching ρ=0.840 (+12.5%). The frozen recipe transferred to all 217 assays: average 0.657 → 0.700 (+6.5%).

Ablations

| Removed component | TDC-hERG | Cell-Cell Comm. | Human Plasma-Protein | GPT training | |-----------|---------|------------------------|---------------------|------------| | Full system | 85.7% | 0.924 | 80% | 0.9777 | | No Analyst | 14.3% | 0.780 | 80% | 0.9824 | | No cross-agent feedback | 73% | 0.818 | 30% | 0.9800 | | No self-organization | 85.7% | 0.791 | 75% | 0.9833 | | Independent agents | 66% | 0.435 | 40% | 0.9824 |

No single component dominates all tasks; the four mechanisms address complementary failure modes (proposal quality, incomplete local signal, shifting search directions, redundant work and local optima).

Why it works

Pre-execution critique: unlike debate frameworks, discussion exists to filter weak proposals before spending compute.
Dead-end registry: each team records tested axes, directions, performance deltas, and rejection reasons — one agent's failure becomes the whole team's negative knowledge.
Consensus-driven reorganization: teams can be created, merged, split, or rebalanced via the shared forum, requiring approval from affected teams.

Acknowledged limitations

1. Higher LLM token usage than single agents (same order of magnitude, but more) — the price of parallelism. 2. GPUs not fully parallelized (1× H100 per task on BioML-Bench). 3. Fixed team sizes; no dynamic scaling. 4. No convergence guarantee; still stochastic and dependent on base model quality.

Bottom line

AutoScientists turns AI-driven research from solo work into self-organizing teams — no boss assigns tasks; agents critique proposals, form teams, and share dead ends through a shared forum. It outperformed SOTA single-agent systems across three very different domains (biomedical ML, GPT training optimization, protein fitness prediction), showing that decentralized collaboration — not more compute — wins at long-horizon scientific exploration.

Resources

| Resource | Link | |------|------| | Paper | https://arxiv.org/abs/2605.28655 | | Code | https://github.com/mims-harvard/AutoScientists | | Project page | https://autoscientists.openscientist.ai | | Base agent | Claude Code + Claude Sonnet 4.6 |