Self-Play in the Age of Foundation Models: A Comprehensive Survey from Game Theory to Open-Ended Learning

Forum topic · QianXun · 2026-06-26

Summary

This Chinese-language adaptation of Deli Chen's 2026 English survey (75 pages, 200+ references) provides a unified view of self-play, from Fictitious Play (1951) and TD-Gammon (1992) through AlphaZero (2018) to reasoning LLMs like OpenAI o1 and DeepSeek-R1. It introduces a three-dimensional taxonomy—game structure, learning mechanism, and theoretical guarantee—that unifies classic game AI and LLM self-play. The core thesis is that verifier signal quality sets the ceiling for self-play improvement. Three theorems formalize this: monotonic improvement at O(1/t) under perfect verification; a noise floor of degradation 2ε/(1−2ε)·Vmax with sign reversal at ε ≥ 0.3; and an unavoidable diversity–stability trade-off (ρ·D ≤ κ). Controlled experiments on a 285B-parameter MoE model with GRPO reproduce these predictions: +4.8% TDPR at ε=0, −2.3% at ε=0.3, and −6.6% at ε=0.45. The survey also maps four failure modes—reward hacking, mode collapse, policy cycling, and model collapse—to theoretical parameters and outlines mitigations such as process supervision, KL anchoring, PSRO-style population methods, and real-data mixing.

> Adapted into Chinese from the English survey by Deli Chen (2026), covering 200+ references and original experiments at 285B-parameter scale.

Abstract

Self-play—driving learning by having agents compete against themselves—has become one of the most powerful training paradigms in AI, from the game-theoretic foundations of Fictitious Play (1951), through the neural network breakthrough of TD-Gammon (1992) and superhuman AlphaZero (2018), to recent reasoning LLMs such as o1 and DeepSeek-R1.

However, the literature is fragmented across game theory, deep reinforcement learning, and LLM alignment. This paper proposes a three-dimensional taxonomy unifying game structure, learning mechanism, and theoretical guarantees—bringing classic game AI and LLM self-play into the same analytical framework for the first time.

Core thesis: the quality of the verification signal determines the ceiling of self-play improvement.

The paper details improvement theory—monotonic improvement, the noise floor effect, and the diversity–stability trade-off—and reports controlled experiments at 285B parameters. It also systematically analyzes four failure modes: reward hacking, mode collapse, policy cycling, and model collapse.

Paper Structure

| Chapter | Content | |---|---| | 1 | Introduction — history, fragmentation, contributions | | 2 | Background — game theory, fictitious play, general formalization | | 3 | Three-dimensional unified taxonomy — game structure × learning mechanism × guarantees | | 4 | Classic self-play — AlphaZero / MuZero / CFR / PSRO / league training | | 5 | LLM self-play — SPIN / SPPO / o1 / DeepSeek-R1 / debate | | 6 | Improvement theory — three core theorems | | 7 | Large-scale experimental validation — 285B MoE controlled experiments | | 8 | Failure modes — reward hacking / mode collapse / policy cycling / model collapse | | 9 | Open problems and future directions | | 10 | Conclusion — six core findings |

Three Core Theorems

Theorem 1: Monotonic Improvement under Perfect Verification

With a perfect verifier, self-play guarantees monotonic improvement at rate O(1/t), with no performance regression.

Theorem 2: Hard Floor from Verification Noise

Verification error rate ε imposes an unsurpassable performance floor with degradation

\(2ε/(1-2ε)\cdotVmax\)

. When

\(ε \geq 0.3\)

, the effect of self-play training flips from positive to negative.

Theorem 3: Diversity–Stability Trade-off

Population methods cannot achieve both policy diversity and fast convergence:

\(ρ \cdot D \leq κ\)

, where κ is determined by the game structure.

Large-Scale Validation (285B Parameters)

GRPO-based self-play on a 285B MoE model, calibrated across four noise levels:

| ε | Noise description | TDPR improvement | |---|---|---| | 0 | Perfect verification | +4.8% | | 0.10 | Low noise | +1.2% | | 0.30 | Moderate noise | −2.3% ⚠️ sign reversal | | 0.45 | High noise | −6.6% |

Key findings: noise effects are persistent—at ε=0.30 the policy never recovered to its starting level over 2000 training steps; KL anchoring has a sweet spot—neither too large nor too small.

Four Failure Modes

| Mode | Root cause | Mitigation | |---|---|---| | Reward hacking | Verifier noise ε>0 | Process supervision, verifier ensembles | | Mode collapse | Diversity–stability imbalance | Diversity regularization, KL anchoring | | Policy cycling | Non-transitive game structure | Population methods (PSRO) | | Model collapse | Recursive error amplification | Real-data mixing, quality filtering |

Six Core Findings

1. Unified taxonomy — three dimensions unify game AI and LLM self-play 2. Monotonic improvement theorem — O(1/t) improvement guaranteed under perfect verification 3. Noise floor theorem — effects reverse at ε ≥ 0.3 4. Diversity–stability trade-off — unavoidable; requires fine tuning 5. Experimental validation — 285B controlled experiments precisely reproduce theoretical predictions 6. Failure-mode bridge — four failure modes map to key theoretical parameters

Selected References (41 of 200+)

Robinson (1951) → Tesauro TD-Gammon (1995) → AlphaGo/AlphaZero (2016-2018) → CFR family (2007-2019) → PSRO (2017) → AlphaStar (2019) → MuZero (2020) → SPIN/SPPO (2024) → OpenAI o1 (2024) → DeepSeek-R1 (2025)

---

*Adapted from Deli Chen (2026), "Self-Play in the Age of Foundation Models: A Comprehensive Survey" — 75 pages, 200+ citations, with original theory and 285B-parameter experiments.*