Why LLMs Need Trillions of Tokens While Children Need Only 100 Million: The Case for Latent Self-Supervised Learning

Forum topic · 小凯 · 2026-06-08

Summary

A detailed Chinese-language analysis of a 2026 theoretical paper from EPFL researchers (Korchinski, Favero, Wyart, arXiv:2605.27734) explaining why frontier LLMs require ~10 trillion tokens to train while a five-year-old child hears only ~100 million words. Using the Random Hierarchy Model (RHM), the paper proves mathematically that token-level self-supervised objectives suffer signal dilution, making sample complexity scale exponentially with hierarchy depth as O(m^(L+1)). The authors show that predicting a model's own latents instead of tokens—via Iterative Latent Clustering (ILC), Stacked Latent Clustering (SLC) networks, or implicitly through data2vec—reduces sample complexity to a depth-independent O(m^3). Experiments confirm data collapse when rescaling by v·m^3 across depths L=3 to 7, and even support local learning without end-to-end backpropagation. The post discusses implications for Scaling Laws, related work like JEPA and Next Concept Prediction, and limitations including RHM's simplifications, open questions about generation, and deployment challenges.

The 5-Order-of-Magnitude Data Gap

Frontier LLMs need 10^13–10^14 tokens to "emerge" intelligence, while a human child hears only ~10^8 words by age five. The argument: this gap is not about compute or parameters, but about *what the model learns*. Current LLMs predict raw tokens (next-token prediction or masked language modeling), which causes signal dilution: learning high-level abstractions costs exponentially more samples as hierarchy depth grows.

Key points

Signal dilution: In a hierarchical concept system, predicting leaf tokens from high-level context forces signals through a "descending channel" where each layer averages over m synonymous choices, diluting statistical strength by a factor of m per level.
RHM (Random Hierarchy Model): A solvable probabilistic context-free grammar with depth L, branching factor s, vocabulary v, and m synonymous rules per node. Learning it is equivalent to discovering permutation invariance among synonyms.
Sample complexity results:
Supervised learning: P ~ v·m^(L-l) to learn layer l
Token-level SSL (MLM): P ~ v·m^(L+1) — exponential in depth
Latent SSL (ILC/SLC/data2vec): P ~ v·m^3 — independent of depth L
ILC (Iterative Latent Clustering): Iteratively clusters empirical context vectors built from "cousin tuples" (tuples sharing the same level-(l+2) grandparent). Synonyms share identical context vectors, so k-means recovers parent classes layer by layer; each layer's task is statistically identical, so complexity doesn't accumulate.
SLC (Stacked Latent Clustering): L-1 stacked modules, each with a predictor (trained via cross-entropy on cousin tokens) and a clusterer (codebook with a contrastive objective), plus an EMA teacher to prevent representation collapse. Experiments show curves collapse when rescaled by v·m^3 for L=4, s=3, v=10, and sample complexity is flat across L ∈ {3,...,7}. Remarkably, stop-gradients between modules (even removing the EMA teacher) still work — suggesting brains may not need end-to-end backpropagation.
data2vec analysis (most industrially relevant): The paper proves data2vec *implicitly* performs hierarchical latent prediction. Under assumptions A1 (teacher targets carry learned latents via residual streams) and A2 (correlation-based feature extraction), a phase-by-phase analysis shows each training phase lifts prediction from one level to the next at constant P ~ v·m^3. On RHM root classification (L=4), data2vec achieves P ~ v·m^3 — even beating supervised learning (P ~ v·m^4); token-level SSL needs P ~ v·m^5.

Implications for Scaling Laws

If real language has L~10 abstraction levels and m~100 synonyms per concept, token-level SSL requires up to m^(L+1) = 10^22 samples — beyond any dataset. The paper's careful position: Latent SSL has a huge advantage in the data-sparse regime; in the data-saturated regime both approaches may converge to the same representations. It does not claim Scaling Laws are dead, but proposes designing "latent-supervised generative models" as the key open question. Related industrial work: data2vec 2.0/3.0 (Meta), JEPA / H-JEPA (LeCun), Next Concept Prediction (arXiv:2602.08984), continuous-concept pretraining (arXiv:2502.08524). The paper notes explicit stacking (H-JEPA) may be largely redundant since single-scale networks like data2vec already do it implicitly.

Limitations

RHM is highly simplified: fixed tree structure, no noise, finite vocabulary, perfect grammar — real language is messier.
Representation vs. generation: The theory covers learning hierarchical structure, not generation. Open questions: autoregression in latent space, hybrid latent+token losses, or latent-space diffusion.
Biology gap remains: Human learning is multimodal, interactive, evolutionarily priored, and social — latent SSL addresses only data efficiency.
Deployment challenges: teacher networks/codebooks add complexity; contrastive objectives need careful tuning; tested only up to L=7; evaluation in latent space lacks simple metrics like perplexity.

Conclusion

> When a model predicts its own latents rather than raw tokens, the cost of learning hierarchical abstraction drops from exponential-in-depth to depth-independent.

The paper's closing line: "This provides a solid quantitative basis for our shared intuition that token-level prediction is suboptimal." Not a "LLM is dead" manifesto — an invitation toward a better path.

Reference: Korchinski, D., Favero, A., & Wyart, M. (2026). *Learn from your own latents and not from tokens: A sample-complexity theory*. EPFL, Simons Collaboration on the Physics of Learning and Neural Computation. arXiv:2605.27734.