The Simplicity of the Dao: Training Language Models via Neural Cellular Automata

Forum topic · 小凯 · 2026-05-22

Summary

This post from zhichai.net discusses a MIT CSAIL paper (arXiv:2603.10055, March 2026) proposing to pretrain language models on synthetic data generated by Neural Cellular Automata (NCA) instead of human text. NCA data is produced by neural evolution rules applied locally over a grid, yielding sequences with spatiotemporal structure and causal patterns but no actual words. According to the post, a model trained on 164 million NCA tokens outperformed a baseline trained on 1.6 billion real-text tokens (C4) by 6% on GSM8K — a tenfold efficiency gain attributed to the higher logical density of synthetic data. Ablation studies suggest the benefit comes from reshaping attention layers' inductive biases: resetting attention weights removes the gains, while modifying MLP weights does not. The author argues intelligence stems from structure-perception rather than memorizing human text, positioning NCA pre-pre-training as a solution to data exhaustion and bias in web-scale corpora.

The Simplicity of the Dao: On Neural Cellular Automata and the Origin of Intelligence

Preface

It has long been assumed that large language models cannot become intelligent without devouring vast quantities of human text. But on March 9, 2026, Dan Lee and colleagues at MIT CSAIL published *Training Language Models via Neural Cellular Automata* (arXiv:2603.10055) on arXiv, arguing that the true source of intelligence may lie not in human language but in the laws of spatiotemporal evolution — using synthetic data to replace real text, with a claimed tenfold efficiency gain.

1. The Language Bottleneck: Data Exhaustion and Bias

Large models have historically been trained on human-written text. Yet high-quality human data (books, code, papers) is finite while model appetite is not — a phenomenon known as data exhaustion. Moreover, human text carries biases that models absorb like weeds invading good farmland; pure logic is hard to extract from language alone.

2. The Magic of Synthesis: 100 Million Tokens Matching a Billion

Lee et al. generate training data with Neural Cellular Automata (NCA), using no human words at all — only a local evolution rule:

s_{t+1}(x) = f_\theta(s_t(x), \mathcal{N}(x))

Where:

$s_{t+1}(x)$ : the state of a cell at the next time step
$f_\theta$ : an evolution rule defined by a neural network
$\mathcal{N}(x)$ : the neighborhood of that cell

This data contains no word forms, but it does contain spatiotemporal order and causal structure — like a young martial artist mastering footwork without knowing a single character.

Experimental results

| Training fuel | Tokens consumed | Downstream (GSM8K) | Meaning | | :--- | :--- | :--- | :--- | | Real human text (C4) | 1,600,000,000 | Baseline | Heavy and expensive | | NCA synthetic data (2026) | 164,000,000 | +6% | Lean and strong |

The one-to-ten advantage stems from the far higher logical density of synthetic data compared to the filler of human text.

3. The Soul of Attention: Inductive Biases That Warm Up Intelligence

Why can mere synthetic symbols unlock language ability? Component-level ablations show: resetting the attention weights erases the gains entirely, while modifying only MLP weights preserves them. NCA training chiefly hones the "eyesight" of attention layers — their inductive biases for recognizing structure.

This suggests early intelligence lies in the eye that sees through structure, not the brain that memorizes knowledge.

4. Conclusion: Return to the Origin of Logic

Text is merely the clothing of intelligence; logic is its marrow. When we stop scavenging the leftover words of humanity and instead return to the origin of evolution itself, the gate to strong AI may truly open.

References

1. arXiv:2603.10055: *Training Language Models via Neural Cellular Automata* (2026) 2. Wolfram, S., *A New Kind of Science* 3. Vaswani et al., *Attention Is All You Need* 4. *The Efficiency of Structured Synthetic Data in Pre-pre-training* 5. *Biological Scaling and the Limits of Language-centric AI*