Auctopus: Multi-Agent AI Runs a Full Autonomous Quantum Physics Experiment

Forum topic · 小凯 · 2026-10-31

Summary

Researchers at Institute of Science Tokyo built Auctopus, a multi-agent AI system that autonomously carried out a complete research project on nitrogen-vacancy (NV) center ensembles in diamond: proposing topics, proving theory, designing experiments, collecting data, and writing conclusions (arXiv 2610.40589, MIT-licensed). The system, roles assigned to models including GPT and Claude variants, institutionalizes Merton's 'organized skepticism' through separated contexts for proposal and review, explicit decision gates, pre-registration (predictions and acceptance criteria hashed before analysis), and an independent reviewer. Notable episodes: the PI rejected 75 proposals; it vetoed an experimental design that would have been satisfied by Fourier interpolation; theorem proved that L²+1 calibration measurements suffice to predict responses of any phase-controlled pulse sequence; and 252,620 exact-arithmetic certificates were generated on the TSUBAME 4.0 supercomputer. A first run passed all ten precision criteria but failed a control—the 2.9% signal jump at 11:48 was traced to a lab member turning on a light. The PI ruled the run inconclusive; a redesigned six-hour run (21,600 records) confirmed the locked prediction (39.7±1.3 vs. 40.0±0.4 ×10⁻³), with interpolation baselines at zero. Limitations include human-built hardware, human-set safety limits, patched proposal diversity, and n=1.

Overview

In 2024, Sakana AI's "AI Scientist" produced a paper whose claimed performance gains rested on flawed experiments—raising the question: once AI can run the entire research pipeline, who guarantees the conclusions hold? Four researchers at Institute of Science Tokyo (Science Tokyo) confronted this directly with Auctopus, a multi-agent system that performed a full research cycle on nitrogen-vacancy (NV) center ensembles in diamond: topic selection, theory, experimental design, data collection, analysis, and conclusions. Paper: arXiv:2610.40589; code MIT-licensed at github.com/E-zClap/auctopus.

Why a diamond

NV centers in diamond trap an electron spin: optically readable with 532 nm green light, coherently controllable with ~2.75 GHz microwave pulses. An *ensemble* of billions of centers with varying detunings and driving strengths yields an averaged response—an ideal real-world testbed that is both physically demanding and automatable.

Institutionalizing skepticism

Translating Merton's "organized skepticism" into architecture, six roles:

Proposers — brainstorm research directions
PI (GPT-6 Pro) — reviews, arbitrates, enforces quality
Computational researcher — independently re-verifies theory, runs supercomputing
Experimentalist (Claude agent on the instrument PC) — data acquisition
Manager — coordination and logging
Independent reviewer — fresh-context review of the final manuscript only

Plus a safety agent enforcing human-set voltage/power limits. Four pillars: (1) separated production/review contexts, (2) mandatory explicit review gates, (3) predictions and acceptance criteria locked before data analysis (pre-registration), (4) rejected proposals may be revised and resubmitted.

Key points

Topic selection: ~30 hours, 75 proposals rejected; the first 17 batches were all rejected. The manager found 52 of 56 new proposals collided with previously rejected families and patched instructions to exclude them.
The winning question: what is the minimum number of calibration measurements needed to predict an inhomogeneous spin ensemble's response to an arbitrary phase-controlled pulse sequence?
The theorem: L²+1 calibrations are both necessary and sufficient, independent of the unknown error distribution (L=3: 10 instead of the naive upper bound of 91). Sufficiency follows because all length-L sequence responses span only an L²+1-dimensional space; necessity is proven by constructing two indistinguishable-but-different ensembles.
Independent verification: symbolic re-derivation, dimension rebuilds (26 for L=5, 37 for L=6), and 252,620 exact rational-arithmetic certificates on TSUBAME 4.0 showing robustness to hidden ~1.2° rotation-axis perturbations.
A crucial veto: the PI rejected a design in which Fourier interpolation could also hit the withheld targets—"being able to predict" does not "prove the law." The revised cross-length design used 10 calibrations and proved both interpolation baselines have exactly zero contrast on the withheld targets.
Pre-registration: all prediction rules and 10 accuracy criteria plus controls were hashed before any withheld data was analyzed.

The 11:48 light

The first 36-sub-run acquisition passed all ten criteria but failed the control. The PI ruled it *inconclusive* despite the perfect scorecard. Investigation found a 7.6% baseline drift and a 2.9% jump at 11:48—a lab member turned on a light. Two program variants sampling the drifting signal at different times produced the systematic offset. Criteria were not relaxed; the run was not rehabilitated. A redesigned run with phase-switching inside a single program, then a final randomized six-hour acquisition, was performed.

Final result

From 21,600 one-second records: locked prediction (39.7±1.3)×10⁻³ vs. measured (40.0±0.4)×10⁻³ (difference under ⅓ of a standard deviation); interpolation baselines predicted 0, observed signal significant at p ≤ 0.001. The paper's wording: the measurement supports the scaling law—"support," not "prove."

What humans did

Only two things: built the instrument, and configured the safety agent's hard limits. AI use was fully disclosed per the Belmont framework; human authors take full responsibility. Stated limitations: human-built hardware, human-set red lines, proposal diversity requiring patches, sample size of one (an existence proof), a reviewer who sees only the manuscript, and substantial compute cost.

Takeaway

Auctopus's answer to "who guarantees the conclusions" is deliberately old-fashioned: institutionalize criticism—context-isolated review, explicit decision gates, pre-registration, and controls that always give competing hypotheses a fair chance. As Feynman said in 1974: the first principle is not to fool yourself—and you are the easiest person to fool. Even when the error comes from a light switch.

---

References: "Collaborative AI agents for autonomous experimental research with solid-state quantum spins" — E. M. Mansouri, E. Kimura, M. Koga, K. Arai (Institute of Science Tokyo), arXiv:2610.40589, submitted 2026-10-29. Code: https://github.com/E-zClap/auctopus