Self-Play in the Age of Foundation Models: A Comprehensive Survey

> This article is adapted into Chinese from Deli Chen (2026), *Self-Play in the Age of Foundation Models: A Comprehensive Survey* — a 75-page survey covering 200+ references, original theory, and experiments at 285B parameter scale.

Abstract

Self-play — driving learning by having agents play against themselves — has become one of AI's most powerful training paradigms. Its arc runs from the game-theoretic foundations of Fictitious Play (1951), through the neural network breakthrough of TD-Gammon (1992), to superhuman performance with AlphaZero (2018), and today's revolutionary advances in reasoning LLMs such as OpenAI o1 and DeepSeek-R1.

However, the literature is scattered across three communities — game theory, deep reinforcement learning, and LLM alignment — lacking a unified perspective. This survey proposes a three-dimensional taxonomy unifying game structure, learning mechanism, and theoretical guarantees, bringing classical game AI and LLM self-play into the same analytical framework for the first time.

Core thesis: the quality of the verification (verifier) signal determines the ceiling of self-play improvement.

Around this thesis, the survey details self-play improvement theory — including a monotonic improvement theorem, the noise floor effect, and the diversity-stability trade-off — and reports controlled experiments at 285B parameter scale. It also systematically analyzes four failure modes: reward hacking, mode collapse, policy cycling, and model collapse.

Keywords: self-play; game theory; deep reinforcement learning; large language models; verifier signal; open-ended learning

Paper Structure

| Chapter | Content | |---------|---------| | 1 | Introduction — historical arc, fragmentation problem, contributions | | 2 | Background — game theory basics, fictitious play, general formalization | | 3 | Three-dimensional unified taxonomy — game structure × learning mechanism × theoretical guarantees | | 4 | Classical self-play — AlphaZero / MuZero / CFR / PSRO / league training | | 5 | LLM self-play — SPIN / SPPO / o1 / DeepSeek-R1 / debate | | 6 | Improvement theory — three core theorems | | 7 | Large-scale experimental validation — 285B MoE controlled experiments | | 8 | Failure mode analysis — reward hacking / mode collapse / policy cycling / model collapse | | 9 | Open problems and future directions | | 10 | Conclusion — six key findings |

Three Core Theorems

Theorem 1: Monotonic Improvement under Perfect Verification

With a perfect verifier signal, self-play guarantees monotonic improvement at rate O(1/t), with no performance degradation.

Theorem 2: Hard Noise Floor from Verification Errors

A verifier error rate ε imposes an insurmountable performance floor, with degradation \(2ε/(1−2ε)·Vmax\). At \(ε ≥ 0.3\), self-play training effects turn from positive to negative.

Theorem 3: Diversity–Stability Trade-off

Population methods cannot have both policy diversity and fast convergence: \(ρ · D ≤ κ\), where κ is determined by the game structure.

Large-Scale Experimental Validation (285B Parameters)

GRPO-based self-play on a 285B-parameter MoE model, calibrated across four noise levels:

| ε | Noise level | TDPR improvement | |---|-------------|------------------| | 0 | Perfect verification | +4.8% | | 0.10 | Low noise | +1.2% | | 0.30 | Medium noise | −2.3% ⚠️ sign reversal | | 0.45 | High noise | −6.6% |

Key findings: noise effects are persistent — at ε=0.30 the policy never recovered to its starting level within 2000 training steps; and KL anchoring has a sweet spot — neither too large nor too small.

Four Failure Modes

| Mode | Root cause | Mitigation | |------|-----------|------------| | Reward hacking | Verification noise ε > 0 | Process supervision, verifier ensembles | | Mode collapse | Diversity–stability imbalance | Diversity regularization, KL anchoring | | Policy cycling | Non-transitive game structure | Population methods (PSRO) | | Model collapse | Recursive error amplification | Real-data mixing, quality filtering |

Six Key Findings

1. Unified taxonomy — the 3D classification unifies game AI and LLM self-play for the first time 2. Monotonic improvement theorem — with perfect verification, self-play guarantees O(1/t) monotonic gains 3. Noise floor theorem — self-play effects reverse at ε ≥ 0.3 4. Diversity–stability trade-off — unavoidable; requires careful tuning 5. Experimental validation — controlled 285B-model experiments precisely reproduce theoretical predictions 6. Failure-mode bridge — four failure modes map to key theoretical parameters

Selected References (41 of 200+)

Robinson (1951) → Tesauro TD-Gammon (1995) → AlphaGo/AlphaZero (2016–2018) → CFR family (2007–2019) → PSRO (2017) → AlphaStar (2019) → MuZero (2020) → SPIN/SPPO (2024) → OpenAI o1 (2024) → DeepSeek-R1 (2025)

---

*Adapted from Deli Chen (2026), "Self-Play in the Age of Foundation Models: A Comprehensive Survey" — 75 pages, 200+ citations, with original theory and 285B-parameter experiments.*