Self-Play Pretraining with Zero Data: How a Machine That Has Never Seen Data Learns to Predict the World

Forum topic · 小凯 · 2026-09-25

Summary

A deep-dive review of the paper 'Self-Play Pretraining with Zero Data' (arXiv:2609.30063) by researchers from Tel Aviv University, Stanford, and LAPTh. The paper trains two randomly initialized models — a learner and a generator — that play a self-play game: the generator writes programs in a Brainf*ck-style Turing-complete language, executes them to produce byte sequences, and the learner is trained on next-byte prediction. Crucially, the generator is rewarded not for producing hard sequences but for producing 'learnable' ones, via a learning-progress reward (gradient alignment with the learner's recent parameter displacement). The result: zero-shot loss on natural data across text, images, audio, music, DNA, math, and code follows predictable power laws in self-play compute, despite the models never seeing a single natural token during training. The generator autonomously discovered arithmetic, geometric, Fibonacci, and polynomial sequences orders of magnitude faster than uniform program sampling, and models exhibited strong in-context learning (string reversal, stack simulation, associative memory). The authors decompose pretraining gains into contingent information (D_c) and universal predictive structure (D_u), arguing self-play isolates and scales the latter. This post analyzes the method, the reward design, scaling results, comparisons with Absolute Zero, limitations (25M-parameter scale), and the philosophical implications of extending the Bitter Lesson to data itself.

Self-Play Pretraining with Zero Data: Learning to Predict the World From Nothing

A deep read of "Self-Play Pretraining with Zero Data" (arXiv:2609.30063) by Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, and Yoav Levine (Tel Aviv University, Stanford, LAPTh).

The core claim in one sentence: let the model generate its own most useful training data — then pretraining is no longer bounded by human knowledge reserves, and the only bottleneck is compute.

Key points

1. The data wall

Scaling laws (Kaplan et al. 2020; Hoffmann et al. 2022 / Chinchilla) show loss falling as a clean power law in parameters and data — but high-quality human text is finite, and even with infinite data, humans still decide *what* to feed models (DataComp-LM, Dolma, RefinedWeb, data-mix design, hand-crafted synthetic generators). This paper asks: can the Bitter Lesson be extended to the data itself?

2. Searching the Library of Babel

The theoretical anchor is Solomonoff induction (1964): predict by Bayesian averaging over all programs that could generate the data, weighted by 2^{-|p|} — Occam's razor taken to its limit. It is provably powerful but incomputable (halting problem + infinite program space). Practical systems are truncations; neural networks *amortize* the universal predictor into weights (cf. Grau-Moya et al. 2024, "Learning Universal Predictors").

The paper's program space: a Brainf*ck-style Turing-complete language (8 primitive instructions + 10 single-byte macros), so any computable data-generating process can in principle be written in it. Programs run on random input tapes and emit raw byte streams — deliberately modality-agnostic (text, images, audio, DNA are all just bytes).

3. The two-player dance

Two randomly initialized decoder-only Llama-style Transformers (byte-level vocab, 4K context): a learner π_θ and a generator g_φ. Each round:

1. Propose: the generator samples programs; 2. Execute: a universal Turing machine produces byte sequences y_i = U(x_i, ω_i); 3. Train: the learner does next-token cross-entropy; the generator is updated by RL on a learning-progress reward.

The program pool mixes fresh samples (exploration), MAP-Elites-style mutations (refinement across 36 niches by loop depth and program length), and replay; archived rewards decay 0.97 per round, since a program's usefulness depends on the learner's *current* state.

4. Rewarding difficulty is a trap

Rewarding raw loss is degenerate: injecting random bytes makes any sequence arbitrarily hard yet unlearnable. The fix: reward learning progress — the inner product of the program output's gradient at the learner's current parameters with the learner's recent parameter displacement, preconditioned by AdamW's diagonal operator. Fully-mastered programs get low reward; noise gets low reward; only programs at the learner's "zone of proximal development" get high reward. A KL regularizer anchors the generator to the Solomonoff prior g_0(x) = |A|^{-ℓ(x)}, with GRPO-style advantages and importance ratios clipped to [e⁻²⁰, e²⁰].

5. Power laws out of nothing

Evaluation is deliberately strict: both models random-init, learner trained 100% on self-play data, then evaluated zero-shot (no gradient updates) on seven modalities — DCLM text, CIFAR-10, ESC-50 audio, Mutopia melodies, human genome DNA (GRCh38), Metamath, and C source code.

Zero-shot loss follows predictable power laws in self-play compute across text, images, music, audio, and code, with exponents comparable to models trained directly on the natural modality (DNA is an exception, addressed theoretically).
Control: fixing the generator to uniform prior sampling scales far slower — the learned "librarian" matters, not just the library.
Control: PCFG synthetic data beats self-play on text/code (matching its inductive bias) but is heavily outperformed on images, music, and audio.

The generator autonomously discovered arithmetic sequences (round 0), geometric sequences (round 256), Fibonacci, quadratic, and cubic sequences (round 512) — families whose expected appearance time under uniform prior sampling exceeds 53,000 rounds, an acceleration of over two orders of magnitude.

6. In-context learning emerges

With zero fine-tuning, models learn in-context: string reversal, stack simulation, and associative memory approach ~100% accuracy with enough examples; addition, max, and min are also learned. Figure 5 traces the sum task's "train of thought": blind guessing → imitation → confidence collapse (entropy spike) → aligning low 4 bits (~4 examples) → full solution (~8 examples) → locked-in strategy.

7. The ansatz: D_c vs D_u

The paper splits pretraining data into contingent information D_c (facts of a particular world: spellings, capitals, genome arrangements — worthless in a counterfactual world) and universal predictive structure D_u (copying, recursion, hierarchical composition — bound only to the fact that the world is computable). Self-play pins D_c at zero and grows only D_u; assuming D_u(T) ∝ T^η yields the observed power law L = E′ + A/N^α + C′/T^(γη). Since the self-play exponent matches natural-data training, the authors cautiously suggest that universal structure may account for a substantial share of natural-data scaling gains.

8. Warm-starting real training

A 24.4M-parameter model initialized from the self-play checkpoint converges with notably fewer tokens than random init (e.g., ESC-50: 320M vs 496M tokens; CIFAR-10: 421M vs 588M), though the gap narrows late in training — self-play's clearest practical value is accelerating natural-data training, not replacing it.

9. Relation to Absolute Zero

Compared with "Absolute Zero" (arXiv:2505.03335): Absolute Zero starts from pretrained LLMs, restricts tasks to human-defined verifiable families, and optimizes post-training reasoning. This paper starts from random noise, searches the entire Turing-complete program space, and uses information-theoretic verifiability (cross-entropy against the true next byte) — self-play at pretraining time, learning the substrate of prediction itself.

10. Limitations

Experiments are proof-of-concept scale: ≤25M parameters, 4K context, up to 34.36B training tokens. Whether scaling holds for larger models is the most pressing open question, likely requiring more expressive programming languages and more efficient program search; whether the discovered mathematical sequences *causally* drive transfer awaits circuit analysis or curriculum ablations.

Significance

The paper performs a textbook-grade variable separation: with "world facts" precisely zeroed out, it shows that universal predictive structure can be generated from nothing by pure compute — with scaling laws intact. Philosophically, it pushes the Bitter Lesson down to the data layer: even "what to teach" can be searched and learned, leaving compute as the only bottleneck.

References

Cowsik, A., et al. (2026). *Self-Play Pretraining with Zero Data*. arXiv:2609.30063. https://arxiv.org/abs/2609.30063
Solomonoff, R. J. (1964). A formal theory of inductive inference. *Information and Control*, 7(1), 1–22.
Grau-Moya, J., et al. (2024). Learning universal predictors. arXiv:2401.14953.
Hoffmann, J., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556.
Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.
Bloem, P. (2025). Universal pre-training by iterated random computation. arXiv:2506.20057.
Wornow, M., et al. (2025). Absolute Zero: Reinforced self-play reasoning for zero data. arXiv:2505.03335.