Tabula Rasa: Why an AI That Reads No Human Data Can Learn to Predict the World
The Thought Experiment
Imagine an infant locked in an empty room from birth — no books, no windows, no external information. Only a typewriter and a Turing machine. The infant types programs, runs them, and observes the output bytes. Slowly it learns: certain programs produce regular sequences, others produce complex ones. It sets itself a loop — write a program, observe the output, try to predict the next byte, refine its understanding.
Months later, you hand it Shakespeare text, a cat photo, or a Bach melody and ask it to predict the next byte. It has never seen such data, yet it predicts far better than random — and its accuracy improves with more compute, following a predictable power law.
This is not science fiction. It is the experimental result of a paper posted to arXiv in September 2026.
Data Is the Bottleneck — But What in Data Is Actually Useful?
LLM progress has largely been a story of feeding more data: Common Crawl, RedPajama, DCLM, FineWeb. But high-quality human data is running out, and synthetic data faces a chicken-and-egg problem: good synthetic data requires a good model, which requires good data.
The authors (Stanford, Tel Aviv University, LAPTh — including Noah D. Goodman and Yoav Levine) asked a more fundamental question: what in human data is actually useful? Their decomposition:
- Contingent information: facts about our specific world — what Shakespeare wrote, what cats look like. If the universe's history had differed, this would be entirely different.
- Universal predictive structure: cross-modal, cross-domain regularities — patterns, recursion, arithmetic, contextual inference. Not about *our* world, but about what *predictable data looks like*.
- Generator: writes programs in a Brainf*ck-style Turing-complete language (8 basic instructions plus 10 macros). Executed programs output byte sequences.
- Learner: standard next-token prediction (cross-entropy) on those bytes.
- Power-law zero-shot transfer: learner loss on natural data across text (DCLM), images (CIFAR-10), music, audio, speech, and code falls with self-play compute along predictable power-law curves.
- Controlled comparisons: uniform sampling from the same program space scales far more slowly; PCFG pretraining transfers well on language-like domains but inconsistently elsewhere. Self-play is the only method with consistent power-law scaling across all modalities. The generic program space alone is not enough — the adaptive curriculum matters.
- Arithmetic sequences (constant first differences)
- Quadratic sequences (constant second differences)
- Cubic sequences (constant third differences)
- Fibonacci-style sequences: \(a_n = a_{n-1} + a_{n-2}\)
- Geometric sequences: \(a_n = r \cdot a_{n-1}\)
- Byte-level tokenization (256 vocabulary, no BPE, no modality-specific tokenizers): truly modality-agnostic — text, image, audio, DNA, code are all byte sequences. Cost: longer sequences.
- Program pool design: each round mixes newly sampled programs, high-reward mutations, and historical replay — elite-preservation resembling genetic algorithms, preventing forgetting.
- Forward-mode automatic differentiation for the gradient-alignment reward (a scalar output gradient over parameters), accelerated with a jvp_flash_attention library.
- Scale limits: experiments capped at <25M parameters, 4K context.
- Small model scale; whether power laws hold at larger scale is unknown.
- A Brainf*ck-style program language; richer languages may be more effective.
- Contingent information still requires real-world interaction; self-play cannot supply it.
- Whether self-play search remains efficient at scale is open.
- Title: Self-Play Pretraining with Zero Data
- Authors: Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine
- Institutions: Stanford, Tel Aviv University, LAPTh
- arXiv: 2609.30063
- Date: September 24, 2026
Standard pretraining mixes both. The paper asks: if you train only on universal predictive structure, with zero contingent information, can a model still learn?
The Generator-Learner Duo
The method, Self-Play Pretraining with Zero Data, pairs two models:
The key question: how does the generator decide which programs to write? Naively rewarding "hard to predict" fails — inserting random bytes into a predictable sequence makes it unpredictable without adding useful structure.
Instead, the reward is a gradient alignment score: whether the learner's gradient on a sequence aligns with its recent learning direction, measured using the AdamW pre-conditioner matrix as an inner product.
The intuition resembles a good tutor: assign problems neither too easy nor too hard, but *exactly at the edge of the learner's ability, extending what it just learned*.
Tabula Rasa to Power-Law Scaling
Both models start from random initialization — true tabula rasa. No pretrained weights, no human data, no injected knowledge. The only priors: a Turing-complete program space and byte-level tokenization (256-token vocabulary).
Results:
Emergent Mathematical Structure
During training, the generator spontaneously produced five recognizable classes of mathematical sequences:
None were hardcoded or learned from human data. Uniform sampling, given the same number of programs, discovered only arithmetic sequences — never Fibonacci or geometric ones within the tested range. Self-play found Fibonacci as early as round 512 (a conservative estimate, since checkpoints were saved every 256 rounds).
This raises a deep question: is mathematics invented or discovered? If a from-scratch AI "invents" Fibonacci and geometric sequences without any human math education, these structures look like natural attractors in computational space — any sufficiently smart search process finds them.
Emergent In-Context Learning
Self-play pretrained models also developed in-context learning (ICL). Across six ICL tasks (string reversal, stack operations, associative recall, max, min, sum), accuracy rose steadily with more in-context examples — approaching 100% on reversal, stack operations, and associative recall. Uniform-sampling and PCFG baselines showed almost no ICL.
Qualitative analysis of the SUM task revealed interpretable strategy switches — from "copy" to "count" to "sum" — with prediction entropy first rising then falling as the model confirms the correct strategy, strikingly similar to how humans learn new rules.
Pre-Pretraining
A practical experiment: use self-play as pre-pretraining, then initialize standard natural-data pretraining from it. Across text, image, and audio, the self-play-initialized model consistently reaches the same validation loss faster than training from scratch — suggesting the learned universal structures are compatible with, not orthogonal to, natural data.
A New Conceptual Frame: Data-Structure Decoupling
The paper motivates a decomposition with real consequences:
1. Contingent information must come from interaction with the real world — it cannot be synthesized from compute. 2. Universal predictive structure can emerge from compute alone.
Implication: if universal structure is the bottleneck, data is not the bottleneck — compute is. It echoes compiler history: what once required manual curation (data collection, like early hand-optimized compilers) becomes an automated search problem (like LLVM/IR-driven optimization).
Engineering Notes
Limitations and Outlook
The authors are candid:
As a proof-of-concept, the paper shows that universal predictive structure can emerge from pure computation, without any natural data — opening a pretraining path built not on "more data" but on "better self-generated curricula." If it scales, it may mark a paradigm shift from data-driven AI to compute-driven AI.