Cheat Sheet: Google AI Self-Improvement Trilogy (Dream-RSI / ScientistTwo / RRSI)

*All figures verified against the original arXiv papers (2026-10-03).*

• Dream-RSI — arXiv:2609.14858 (Sep 14)
• ScientistTwo — arXiv:2609.19644 (Sep 17)
• RRSI — arXiv:2609.24972 (Sep 21)

The three-sentence takeaway

1. The three papers operate at three different layers: how to explore (search strategy), how to modify itself (the harness), and how to do research (the full pipeline). 2. All three explicitly state that model weights are unchanged. This "recursion" is behavioral, not parametric. 3. The real underlying theme is the third paper: self-improvement itself overfits — the page most press releases skip.

Quick comparison

| | Dream-RSI | RRSI | ScientistTwo | |---|---|---|---| | What it modifies | Exploration strategy (sequential/parallel/stopping) | Prompts / control flow / tools / memory | Hypothesis → experiment → ablation → paper → defense | | Changes weights? | No, explicitly frozen | No, frozen backbone | No | | Mechanism in one line | Historical logs act as an exact-replay simulator; new strategies are trained "in dreams" | Regularizes self-improvement: annealed edit budget + critic + pruner | Six-stage multi-agent pipeline; the defense engine actually runs follow-up experiments | | Headline numbers | Lasso 550 → 317 calls; cost down 1.7–162× | OOD 43.6 vs 39.7; 30% token savings | 86/107 improvements; over three rounds 10.9% → 9.6% → 8.2% | | Stance | Constructive | Cold water | Constructive |

Auditing the four headline numbers

| Number | Reality | Verdict | |---|---|---| | 107 problems | ICML 2026 Spotlight 64 / NeurIPS 2025 38 / ICLR 2026 5. Extraction criteria not stated in the main text (Appendix A.1) | Clear sourcing, unclear extraction | | 86/107 = 80.4% | By venue: ICML 76.6% / NeurIPS 86.8% / ICLR 80.0% | Credible | | 25.2% mean gain | Same table's median is only 7.7% — a 3.3× right skew. ICML's 49 papers (57% of the count) drive 78% of the gain mass; back-solving the ICML mean gives 34.6%, and 12.8% with ICML excluded | Split metric | | $3,765 / 2.5 days | Covers only 33 problems; tokens + VM only. Human labor unaccounted; no human-cost baseline | Caution |

Back-solve (this report's inference, not disclosed in the paper): Overall 86×25.2 = 2167.2; NeurIPS 33×13.9 = 458.7; ICLR 4×3.8 = 15.2 → ICML's 49 papers = 1693.3, mean 34.56%. The table conspicuously omits an ICML breakdown column.

Three weaknesses to remember

1. The 25.2% is dominated by a few subsets. No full gain list, no distribution plots, no outlier analysis, no winsorization. 2. ScientistTwo never directly answers "does self-improvement overfit?" RRSI shows harness evolution can cause rank inversion; ScientistTwo's 107 problems are exactly its evolution set (baselines, ideas, and validation all on the same problems), with no OOD validation. 3. Autonomy depends on third-party closed models. Per the paper: Gemini 3.6 Flash + Claude Opus 4.8, acknowledged as proprietary.

The sharpest cuts

• Agent Laboratory's own confession (arXiv:2501.04227): automatic review scored 6.1/10 while humans gave 3.8/10 (a 2.3-point gap); LLM self-agreement was only 53.3%. Autonomous mode fell well below the NeurIPS acceptance mean (the paper lists both 5.85 and 5.9). This is the only third-party-published bias measurement of using LLM reviewers as judges.
• Google's own cut (arXiv:2608.26701, Aug 2026): an extended Co-Scientist reviewed 50 manuscripts from this family and found statistical tests treated as computed but never computed, evaluation scaffolds secretly favoring the proposed method, and uncited reuse. Fabrication rate dropped 90% → 4% after fixes. The same group admitted one month earlier that 90% of manuscripts contained fabrications — yet ScientistTwo's 49/49 zero-violation result was tested on only 49 papers (49 < 86 < 107).

Human blind review: done, but the result is "parity"

ScientistTwo's Table 10: 9 reviewers × 33 papers, 1–5 scale. Notably, "no human blind review" claims must be corrected.

| Dimension | Absolute | vs human papers (3.0 = parity) | |---|---|---| | Introduction | 4.2 | 3.1 (AI wins) | | Methods (rigor) | 4.1 | 2.9 (humans win) | | Experiments (baselines) | 4.0 | 3.5 (AI wins) | | Experiments (ablations) | 4.3 | 3.3 (AI wins) | | Overall (top-conference readiness) | 3.7 | 3.0 (exact parity) |

AI scores on completeness, loses on rigor. The human conclusion is parity, not superiority.

"Surpassing accepted papers" holds on only one ruler

| Subject | ScholarPeer (in-distribution) | Stanford (held-out) | |---|---|---| | ICLR 2026 Accepted | 6.8 / 60.0% | 5.2 / 60.0% | | NeurIPS 2025 Accepted | 6.2 / 65.8% | 5.5 / 76.3% | | ICML 2026 Spotlight | 6.9 / 79.7% | 6.1 | | ScientistTwo | 7.5 / 91.9% | 5.7 / 72.1% |

It wins on the ruler it was tuned against; on an unseen one, the advantage disappears. ScholarPeer was also used to refine draft quality.

Three incomparable definitions of "surpassing humans"

No system performs a human-comparison on the same denominator × same judge:

• AI Scientist v1/v2: acceptance-threshold criterion (3 submissions, 1 pass, top 45%)
• Agent Laboratory: mean-score criterion (1.45–2.05 points below)
• ScientistTwo: self-built LLM-reviewer criterion (91.9% / 72.1%)

The only direct blind comparison against human papers is Table 10 — and its conclusion is parity.

The meta-question: what has been handed over, and what remains human

• Handed over: execution and iteration — how to search, modify the harness, run experiments, respond to critiques, write papers.
• Still human: judgment and accountability — what problems to pose, how to sample, who judges truth, who answers for conclusions.

The acceptance step is the subtlest: 91.9% was achieved on an in-distribution LLM reviewer, while genuine human blind review says "parity with humans" — and AI loses on methodological rigor. Judgment has formally been handed to models but not substantively to humans.

One-line memorable quote

> This is not a victory declaration that "AI has replaced researchers," but an experiment in which humans have finally handed evaluation discipline over to AI as well. The outcome is undecided; the bill has not been settled.

---

*Parent document: Google self-improvement trilogy deep research report (same directory) · five-way verification VERDICT=PASS. The ICML mean of 34.56% and the "12.8% excluding ICML" figure are arithmetic inferences from Tables 3/4, not directly disclosed in the original paper.*