AutoScientists: Self-Organizing AI Agent Teams for Long-Running Scientific Experimentation

Forum topic · 小凯 · 2026-06-03

Summary

A Harvard research team (Shanghua Gao, Ada Fang, Marinka Zitnik) introduces AutoScientists, a decentralized multi-agent framework in which LLM agents self-organize into teams for long-running scientific experimentation without a central orchestrator. Agents coordinate through a shared state containing the champion model, a global experiment log, a structured discussion forum, and team-local queues with dead-end registries. The system alternates Discussion phases, where analyst agents critique and filter proposals before any compute is spent, with Execution phases where experiment agents run propose-execute loops in parallel. On BioML-Bench (24 biomedical ML tasks), AutoScientists reaches an average leaderboard percentile of 74.40%, beating the strongest baseline by 8.33 points. In GPT NanoChat training optimization from scratch, it matches baseline quality with roughly 1.9x fewer experiments, and from a champion checkpoint it finds 7 accepted improvements (val_bpb 0.9730) where the single-agent baseline accepts none. On ProteinGym fitness prediction, it lifts a SOTA model (Kermut) from Spearman rho 0.747 to 0.840 on ACE2-Spike binding, with a frozen recipe generalizing +6.5% across 217 assays. Ablations show the components address complementary failure modes rather than one dominating factor. Paper: https://arxiv.org/abs/2605.28655; code: https://github.com/mims-harvard/AutoScientists.

AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

> Paper: AUTOSCIENTISTS: Self-Organizing Agent Teams for Long-Running Scientific Experimentation > Authors: Shanghua Gao*, Ada Fang*, Marinka Zitnik (Harvard University) > Paper: https://arxiv.org/abs/2605.28655 > Code: https://github.com/mims-harvard/AutoScientists > Project page: https://autoscientists.openscientist.ai

Why a single agent is not enough for scientific research

Scientific research is not a one-shot task. It is a long-running loop of:

hypothesis → experiment design → execution → revision → new hypotheses
parallel exploration of competing hypotheses
dynamic re-prioritization as evidence falsifies or supports directions
remembering failures to avoid repeated mistakes

Existing agent designs fall short:

| Approach | Limitation | |------|------| | Single agent (AIDE, Autoresearch) | Single-trajectory search, no parallel exploration | | Multi-agent with coordinator (PI-Scientist-Critic style) | Central planner fixes the search decomposition upfront | | Debate frameworks | Converge to consensus rather than filtering weak proposals |

Core insight: long-running scientific experimentation calls not for a better single agent or a boss-led multi-agent system, but a decentralized, self-organizing team — agents decide research directions themselves, form teams, critique proposals, and share failure knowledge.

Architecture: a decentralized self-organizing team

No central orchestrator. All agents coordinate via shared state and a shared forum instead of top-down task assignment.
Two alternating phases:
*Discussion Phase*: agents analyze the current champion model p*, propose candidate directions, critique each other's proposals, identify gaps in the search space, and self-organize into K teams.
*Execution Phase*: teams run parallel propose-execute loops until stagnation. After 10 consecutive non-improving experiments, a new Discussion Phase can reorganize teams.
Two agent roles:
*Analyst agents* (default 3): maintain team search knowledge, audit the experiment log for untested directions, rank the proposal queue by observed effect size, and analyze champion updates.
*Experiment agents* (default 6): claim experiments from team queues, apply code changes to p*, train, log results, and re-confirm improvements within the noise band.
Four-layer shared state: champion p* (globally readable), experiment log L (globally readable), shared forum F (structured discussion posts, globally readable), team-local state (queues, dead-end registry D, hypotheses — cross-team readable).

Key design: all results, including failures, are visible to every agent. One agent's dead end instantly becomes the whole team's negative knowledge.

Benchmark results

1. BioML-Bench (24 biomedical ML tasks)

| System | Avg. leaderboard percentile | |------|------| | AutoScientists | 74.40% (+8.33% vs strongest baseline) | | Autoresearch | 66.07% | | Biomni | lower |

Largest gains in drug discovery: 46.16% → 64.52%
Protein engineering saturated: both reach 96.97%
Completed all 24 tasks

2. GPT NanoChat training optimization

From scratch: AutoScientists reaches val_bpb ≈ 0.978 in 34 experiments vs 65 for Autoresearch — a 1.9x efficiency gain.
From the champion checkpoint (val_bpb = 0.9777): AutoScientists finds 7 accepted improvements in 93 experiments → 0.9730; Autoresearch accepts 0 of 100 attempts.
Notably, AutoScientists' first discovered improvement (query-key normalization order) was never proposed in Autoresearch's 100 attempts — a broader hypothesis space, not just more compute.

3. ProteinGym fitness prediction

Starting from SOTA model Kermut (ACE2-Spike binding Spearman ρ = 0.747)
Discovers: three-GP ensemble + extended zero-shot features + greedy diversity feature selection + quantile-warping objective → ρ = 0.840 (+12.5%)
Frozen recipe applied to all 217 assays: average 0.657 → 0.700 (+6.5%)

Ablations

| Removed component | TDC-hERG | Cell-Cell Comm. | Human Plasma-Protein | GPT training | |-----------|---------|--------|-----|------------| | Full system | 85.7% | 0.924 | 80% | 0.9777 | | No Analyst | 14.3% | 0.780 | 80% | 0.9824 | | No Cross-Agent Feedback | 73% | 0.818 | 30% | 0.9800 | | No Self-Organization | 85.7% | 0.791 | 75% | 0.9833 | | Independent Agents | 66% | 0.435 | 40% | 0.9824 |

No single component dominates everywhere. The four mechanisms solve complementary failure modes: analysts address proposal-quality bottlenecks, cross-agent feedback addresses incomplete local signals, self-organization addresses shifting search directions, and shared state prevents duplicated work and local optima.

Why it works

1. Pre-execution critique: unlike debate frameworks that converge to consensus, AutoScientists uses discussion to filter out weak proposals before spending experimental compute. 2. Dead-end registry: each team logs tested axes, directions, performance changes, and rejection reasons — one agent's failure becomes the team's negative knowledge. 3. Consensus-driven team reorganization: agents can propose creating, merging, splitting, or rebalancing teams via the forum, requiring approval from affected teams.

Acknowledged limitations

1. Higher LLM token usage than a single agent (same order of magnitude, but more) — the price of parallelism. 2. GPUs not fully parallelized: BioML-Bench evaluation limited to 1×H100 per task. 3. Fixed team size: no dynamic scaling. 4. No convergence guarantee: still stochastic and dependent on base model quality.

Bottom line

AutoScientists turns AI-driven research from solo work into a self-organizing team — no boss assigns tasks; agents critique proposals, form teams, and share dead-end logs through a shared forum. It outperforms SOTA single-agent systems across three very different scientific domains (biomedical ML, GPT training optimization, protein fitness prediction). This is not a victory of more compute, but of better organization.

Resources

| Resource | Link | |------|------| | Paper | https://arxiv.org/abs/2605.28655 | | Code | https://github.com/mims-harvard/AutoScientists | | Project page | https://autoscientists.openscientist.ai | | Base agent | Claude Code + Claude Sonnet 4.6 |