MIT CSAIL Proposes Pretraining Language Models with Neural Cellular Automata Instead of Human Text

Forum topic · 小凯 · 2026-05-22

Summary

A zhichai.net forum post discusses an MIT CSAIL paper (arXiv:2603.10055) by Dan Lee, Seungwook Han, Akarsh Kumar, and Pulkit Agrawal proposing pre-pretraining language models on synthetic, non-linguistic data generated by Neural Cellular Automata (NCA). NCA grids evolve via a shared neural network rule, producing structured spatiotemporal sequences with Zipf-like statistics. Reported results: NCA pre-pretraining on only 160M tokens improves OpenWebText perplexity by 5.7-8.6% and speeds convergence 1.4-1.6x, outperforming pre-pretraining on 1.6B tokens of real C4 text. Gains transfer to web text, math (OpenWebMath), and code (CodeParrot), with BigBench-Lite pass@4 reaching 36.5% vs 29.7% for C4. Component ablations show attention layers carry the transferable computation, while optimal NCA complexity varies by target domain. The post argues this could ease data exhaustion and enable task-tailored synthetic pretraining.

This forum post introduces a recent MIT CSAIL paper: *Training Language Models via Neural Cellular Automata* (arXiv:2603.10055) by Dan Lee, Seungwook Han, Akarsh Kumar, and Pulkit Agrawal. Its central claim: the source of intelligence in language models need not be human language — it may lie in the underlying dynamics of spatiotemporal evolution.

The problem: data exhaustion and bias

LLM pretraining depends on human-written text (books, papers, web pages, code), which poses three issues:

Data exhaustion: per Villalobos et al., high-quality human text may run out around 2028 as model appetite grows exponentially while data supply grows linearly.
Bias: human text embeds biases, errors, and noise that models absorb.
Entanglement: natural language mixes factual knowledge with reasoning, forcing models to memorize facts alongside learning inference.

This raises a fundamental question: is natural language the only path to intelligence?

The method: Neural Cellular Automata (NCA)

NCA fuse classical cellular automata (e.g., Conway's Game of Life) with neural networks. Each cell's next state is computed by a neural rule shared across all cells:

s_{t+1}(x) = f_\theta(s_t(x), \mathcal{N}(x))

where $f_\theta$ is the learned update rule, $\mathcal{N}(x)$ the neighborhood states, $$t$$ time, and $$x$$ spatial coordinates. Local interactions alone produce globally complex patterns. The authors use NCA to generate unlimited synthetic, non-linguistic spatiotemporal trajectories whose symbol statistics resemble the Zipf distribution of natural language. Complexity is controllable via alphabet size ( $$n$$ ) and rule complexity.

Key experimental findings

1. Less is more

| Pre-pretraining data | Tokens | OpenWebText perplexity gain | Convergence speedup | | :--- | :--- | :--- | :--- | | None (scratch) | — | baseline | baseline | | Dyck language (control) | 160M | marginal | slight | | C4 real text | 1.6B | baseline-comparable | baseline | | NCA synthetic | 160M | +5.7%~8.6% | 1.4~1.6x |

NCA on 160M tokens beats 1.6B tokens of C4 (10x the compute). Hypothesis: C4 mostly teaches shallow syntax, while NCA directly trains long-range dependency tracking and latent-rule inference.

2. Cross-domain transfer

Perplexity drops of ~4-5% transfer to web text (OpenWebText), math (OpenWebMath), and code (CodeParrot), with convergence up to 1.6x faster. The advantage persists throughout training and often widens — it is not a transient initialization effect.

3. Downstream reasoning gains

| Benchmark | Scratch | C4 | NCA | | :--- | :--- | :--- | :--- | | GSM8K pass@1 | 3.8% | — | 4.4% | | GSM8K pass@32 | 36.6% | — | 37.9% | | HumanEval pass@1 | baseline | — | improved | | BigBench-Lite pass@4 | 25.9% | 29.7% | 36.5% |

Mechanism: attention is the soul

Ablations that selectively reset components after NCA pre-pretraining show:

Attention: largest degradation — carries the most transferable computational primitives (dependency tracking, latent-rule inference).
MLP / LayerNorm: moderate for text, nearly none for code — they encode domain-specific patterns and statistics.
Embedding: relatively minor impact.

Additionally, the optimal NCA complexity differs by target domain: code prefers small alphabets and simple dynamics; math and web text prefer larger alphabets and richer dynamics. Synthetic data distributions can thus be systematically tuned per downstream task.

Broader implications

1. Origins of intelligence: emergent reasoning may depend on the latent structure underlying coherent sequences, not semantic content. Text is a lossy record of human cognition; NCA trains the underlying computation directly. 2. Future of data: if synthetic data can match or beat real data at smaller scale, cost, and with more control, data exhaustion may be solvable — pointing toward a "synthetic pre-training, real fine-tuning" paradigm. 3. Domain adaptation: pretraining need not be one-size-fits-all; corpus "flavors" can be customized per downstream domain.

References

1. Lee, D., Han, S., Kumar, A., & Agrawal, P. *Training Language Models via Neural Cellular Automata*. arXiv:2603.10055. https://arxiv.org/abs/2603.10055 2. Villalobos, P., et al. (2022). *Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning*. 3. Mordvintsev, A., Randazzo, E., Niklasson, E., & Levin, M. (2020). *Growing Neural Cellular Automata*. Distill, 5(2):e23. 4. Vaswani, A., et al. (2017). *Attention Is All You Need*. NeurIPS. 5. Kaplan, J., et al. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361.