Latent Self-Supervision vs. Token Prediction: Why LLMs Need Trillions of Tokens but Children Only Need 100 Million Words

Forum topic · 小凯 · 2026-06-08

Summary

Frontier LLMs require on the order of 10 trillion tokens to train, while a human five-year-old has heard only about 100 million words—a 100,000x data-efficiency gap. A theoretical paper from EPFL researchers (Korchinski, Favero, and Wyart, 2026, arXiv:2605.27734) argues this gap stems not from compute but from what models learn: next-token prediction forces high-level abstractions to be learned through 'signal dilution' across hierarchy levels, making sample complexity grow exponentially with depth, P ~ v·m^(L+1), as formalized in the Random Hierarchy Model (RHM). The paper proves that if models instead predict their own latent variables (latent self-supervised learning), sample complexity drops to P ~ v·m^3—independent of depth. The authors introduce ILC (Iterative Latent Clustering) and SLC (Stacked Latent Clustering) algorithms achieving this bound, and show that data2vec implicitly performs hierarchical latent prediction, matching the O(m^3) complexity empirically. While the paper does not claim Scaling Laws are dead—methods converge in data-rich regimes—it suggests latent-supervised generative models could dramatically improve data efficiency in sparse regimes, connecting to JEPA and related industry research.

AI's Data Hunger: Why GPT Needs Trillions of Tokens but a Human Child Only Needs ~100 Million Words

> Frontier LLMs need ~10 trillion tokens to "emerge" intelligence, while a human five-year-old has heard only ~100 million words. A 2026 paper from EPFL and the Simons Collaboration provides the first mathematical proof that if AI predicts its own latents instead of raw tokens, sample complexity drops from the depth-exponential O(m^(L+1)) to depth-independent O(m^3).

The 5-Order-of-Magnitude Data Gap

| Learner | Data | Capabilities | |---|---|---| | GPT-4-class LLM | 10^13–10^14 tokens | Grammar, reasoning, partial world knowledge | | Human child (age 5) | ~10^8 words | Grammar, basic logic, physical intuition, social rules | | Gap | 100,000x | — |

This cannot be explained by "brains being more complex"—the human brain has ~86 billion neurons while top LLMs have trillions of parameters. The paper's key hypothesis: the problem is not data quantity but what the model learns. Current LLMs train on next-token prediction or masked-token modeling—predicting the *lowest-level* units—causing signal dilution: learning high-level abstractions costs exponentially more with hierarchy depth.

Signal Dilution and the RHM

The paper uses the Random Hierarchy Model (RHM), a probabilistic context-free grammar with depth L, branching factor s, vocabulary v, and m synonymous production rules per node. Learning RHM amounts to discovering synonym-exchange invariances.

Sample complexity comparison:

| Objective | Sample complexity | Depth dependence | |---|---|---| | Supervised | P ~ v·m^(L−l) for level l | Exponential | | Token-level SSL (MLM) | P ~ v·m^(L+1) | Exponential | | Latent SSL (ILC/SLC/data2vec) | P ~ v·m^3 | Independent of L |

Token-level SSL is exponential because signals for level-l latents must propagate down l+2 "descent channels" to predict tokens, each channel diluting the signal by an m factor. With L~10 abstraction levels and m~100 expressions per concept, m^(L+1) = 10^22—far beyond any dataset.

Latent SSL: Breaking the Exponential Curse

Key insight (Figure 1): latents at the same hierarchy level have strong correlations; predicting a leaf token from a high-level context forces the signal through the full descent tree.

ILC (Iterative Latent Clustering)

Initialize h^(0) = x; for each level, build s-tuples, compute empirical context vectors using "cousin tuples" (sharing an l+2-level grandparent), cluster with k-means (k=v) to define the next-level latents.
Theorem 1 (informal): with P ≥ C·[v·m·log(L·v·m/δ) + v·m^3/(1−f)·log(L·v·m/δ)], ILC recovers all non-root levels—since each recovered level makes the next prediction task statistically identical, complexity does not accumulate.

SLC (Stacked Latent Clustering)

L−1 identical modules, each with a Predictor (cross-entropy on cousin-token prediction) and a Clusterer (contrastive codebook mapping), trained with an EMA teacher to prevent representation collapse.
Experiments: for L ∈ {3,...,7}, sample complexity does not change with L; curves collapse when rescaled by v·m^3.
Ablations (Figure 9): SLC works even with stop-gradients between modules and without the EMA teacher—suggesting brains may not need end-to-end backpropagation.

data2vec Already Does This Implicitly

The paper shows data2vec (Baevski et al., 2022), which trains students to predict averaged teacher activations, implicitly performs hierarchical latent prediction via two assumptions: (A1) teacher targets carry already-learned latents through residual paths, and (A2) gradient descent extracts features once signal exceeds sampling noise. Phase-by-phase analysis gives P_data2vec ~ v·m^3. Empirically (Figure 4), root classification required P ~ v·m^3 for data2vec vs. P ~ v·m^5 for token-level SSL—better even than supervised learning (P ~ v·m^4).

Implications for Scaling Laws

If Latent SSL truly reduces sample complexity from O(m^(L+1)) to O(m^3):

| Scenario | Token-level SSL | Latent SSL | |---|---|---| | L=5, m=100 | ~10^12 | ~10^6 | | L=10, m=100 | ~10^22 | ~10^6 | | Realistic (L~10, m~1000) | ~10^33 | ~10^9 |

The paper is careful: it does not declare Scaling Laws dead. Latent SSL has a large advantage in data-sparse regimes; in data-saturated regimes both approaches may converge to the same representations. It proposes controlled comparisons between data2vec and next-token baselines as a first test, and notes that explicit stacking (e.g., H-JEPA) may be largely redundant if single-scale networks like data2vec already perform implicit hierarchical latent prediction. Related work: JEPA (LeCun, 2022), data2vec 2.0/3.0, Next Concept Prediction (arXiv:2602.08984), continuous-concept pretraining (arXiv:2502.08524).

Limitations

RHM is highly simplified: fixed tree structure, no noise, finite vocabulary, perfect grammar—real language is messier.
Representation vs. generation: how latent-only models generate readable text (autoregression in latent space, hybrid losses, latent diffusion) remains open.
The biological gap isn't closed: human learning is multimodal, interactive, prior-structured, and social—data efficiency is only one factor.
Deployment challenges: teacher networks and codebooks add complexity; contrastive objectives can collapse; experiments only reach L=7; evaluation without perplexity is hard.

Conclusion

> When models predict their own latents rather than raw tokens, the cost of learning hierarchical abstractions drops from exponential-in-depth to depth-independent.

The paper's contributions: (1) a mathematical theory of why LLMs need so much data; (2) proof that latent SSL has an exponential theoretical advantage; (3) an explanation of why data2vec already works—it was doing this implicitly; (4) an industrial direction: design latent-supervised generative models. As the authors put it, this provides "a solid quantitative basis for our collective intuition that token-level prediction is suboptimal."

Reference: Korchinski, D., Favero, A., & Wyart, M. (2026). *Learn from your own latents and not from tokens: A sample-complexity theory*. EPFL, Simons Collaboration on the Physics of Learning and Neural Computation. arXiv:2605.27734.