Training Language Models with Neural Cellular Automata: Escaping Human Data Limits

Forum topic · 小凯 · 2026-05-22

Summary

MIT CSAIL researchers (Lee, Han, Kumar, Agrawal) have published arXiv:2603.10055 proposing that large language models could be pre-trained on synthetic, non-linguistic data generated by Neural Cellular Automata (NCA), bypassing the looming exhaustion of high-quality human text. NCA apply a shared local neural update rule across a grid, producing temporally evolving sequences whose statistics resemble Zipfian language distributions. In experiments, NCA pre-pre-training with only 160 million tokens outperformed real-text pre-pre-training on 1.6 billion C4 tokens, improving OpenWebText perplexity by 5.7-8.6% and accelerating convergence 1.4-1.6x. Gains transferred to web text, mathematics, and code, and lifted BigBench-Lite pass@4 from 25.9% (scratch) to 36.5%. Ablations indicate attention layers encode the transferable computation, while MLPs store domain-specific patterns, suggesting synthetic-data complexity can be tuned per target domain.

TL;DR

MIT CSAIL researchers propose pre-pre-training language models on synthetic trajectories produced by Neural Cellular Automata (NCA), local rule-based dynamical systems unrelated to human language. With only 160M synthetic tokens, they match or surpass 1.6B tokens of C4 real text, cut perplexity by up to 8.6%, converge 1.4-1.6x faster, and improve BigBench-Lite pass@4 from 25.9% to 36.5%.

The Problem: Three Limits of Human Text

1. Data exhaustion. High-quality human text is finite; estimates suggest depletion by ~2028. 2. Bias and noise. Human corpora encode human biases that are hard to filter. 3. Factual knowledge entangled with reasoning. Models must memorize trivia alongside learning to reason.

The authors ask: *is natural language the only road to intelligence?*

What are Neural Cellular Automata?

NCA fuse classic cellular automata (e.g., Conway's Game of Life) with neural networks. Each cell updates its state using a shared small neural network $f_\theta$ applied to its own state and its local neighborhood $\mathcal{N}(x)$ :

s_{t+1}(x) = f_\theta(s_t(x), \mathcal{N}(x))

Despite purely local rules and parameter sharing, NCA trajectories exhibit rich global spatio-temporal structure whose token statistics follow a Zipf distribution, mirroring natural language.

Key Empirical Findings

1. Less is more

| Pre-pre-training | Tokens | OpenWebText PPL improvement | Convergence | |---|---|---|---| | None (scratch) | — | baseline | baseline | | Dyck language | 160M | marginal | slightly faster | | C4 real text | 1.6B | ~baseline | baseline | | NCA synthetic | 160M | +5.7% to +8.6% | 1.4-1.6x faster |

NCA wins at 1/10 the tokens and compute of C4. The authors hypothesize C4 mostly trains shallow syntactic patterns, while NCA train long-range dependency tracking and latent-rule inference.

2. Cross-domain transfer

Gains (4-5% perplexity reduction) transfer to:

Web text (OpenWebText)
Mathematics (OpenWebMath)
Code (CodeParrot)

The advantage persists throughout training and often grows, indicating a genuine token-efficiency boost rather than a warm-start effect.

3. Reasoning benchmarks

4.4%

37.9%

36.5%

Where Do the Gains Live? Component Ablations

Resetting components after NCA pre-pre-training reveals a clear division of labor:

| Reset component | OpenWebText effect | CodeParrot effect | Interpretation | |---|---|---|---| | Attention | largest drop | largest drop | Carries transferable computation | | MLP | moderate | minimal | Stores domain-specific patterns | | LayerNorm | moderate | minimal | Stores domain-specific statistics | | Embedding | mild | mild | Less critical |

Insight: attention weights transfer universally; MLP weights can even hurt cross-domain transfer and may need re-initialization.

Tuning Synthetic Complexity per Domain

Optimal NCA complexity differs by target domain:

| Target domain | Optimal NCA | Why | |---|---|---| | Code | low (small alphabet, simple dynamics) | nested logic + delimiters | | Math | high (large alphabet, rich dynamics) | long-range reasoning | | Web text | high | diversity + long-range structure |

This implies synthetic pre-training distributions can be systematically tuned for downstream tasks rather than one-size-fits-all.

Broader Implications

Source of intelligence. Reasoning may depend on the formal structure underlying language, not its semantic content. NCA teach that structure without words.
Data future. Synthetic data may dissolve the data-exhaustion wall, enabling a *"synthetic rehearsal, then real refinement"* paradigm.
Domain-targeted pre-training. Different domains prefer different rule complexity, motivating bespoke synthetic curricula.

References

1. Lee, D., Han, S., Kumar, A., & Agrawal, P. (2026). *Training Language Models via Neural Cellular Automata*. arXiv:2603.10055. https://arxiv.org/abs/2603.10055 2. Villalobos, P., et al. (2022). *Will we run out of data?* (data-exhaustion analysis). 3. Mordvintsev, A., Randazzo, E., Niklasson, E., & Levin, M. (2020). *Growing Neural Cellular Automata*. Distill, 5(2):e23. 4. Vaswani, A., et al. (2017). *Attention Is All You Need*. NeurIPS. 5. Kaplan, J., et al. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361.