Simplicity Is Ultimate: Neural Cellular Automata as a Path to Intelligence's Origins

Forum topic · 小凯 · 2026-05-22

Summary

A Chinese tech forum post discusses a claimed MIT CSAIL paper (arXiv:2603.10055, dated March 2026, authored by Dan Lee et al.) titled 'Training Language Models via Neural Cellular Automata.' The paper argues that genuine intelligence may stem from spatiotemporal evolution laws rather than human language itself. The method uses Neural Cellular Automata (NCA) to generate synthetic pretraining data via a learned local update rule: s_{t+1}(x) = f_θ(s_t(x), N(x)). According to the post, models trained on 164 million NCA synthetic tokens outperformed a baseline trained on 1.6 billion real text tokens (C4) by 6% on GSM8K — roughly a 10x data-efficiency gain. Ablation analysis attributes the benefit to attention layers acquiring structural inductive biases, not MLP memorization. The post frames this as a response to data exhaustion and bias in human text corpora, concluding that logic and structure, not words, form the marrow of intelligence. It references Wolfram's cellular automata work and the Transformer architecture lineage. Note: the arXiv ID and date fall outside verifiable range, so readers should treat specifics as unconfirmed.

Simplicity Is Ultimate: On Neural Cellular Automata and the Origins of Intelligence 🧬

*Translation and summary of a zhichai.net forum post discussing arXiv:2603.10055.*

Prologue

Conventional wisdom holds that large language models must absorb vast quantities of human text to become intelligent. But a March 2026 paper from MIT CSAIL by Dan Lee et al., "Training Language Models via Neural Cellular Automata" (arXiv:2603.10055), challenges this: the true source of intelligence may lie not in human language but in the laws of spatiotemporal evolution. Synthetic data, it claims, delivers a tenfold efficiency gain.

The Language Problem: Data Exhaustion and Bias

Data exhaustion: high-quality human text (books, code, papers) is being consumed faster than it is produced, threatening a stall in AI progress.
Human-written text carries biases; models trained on it inherit them. Pure logic is hard to extract from natural language.

The Synthetic Method: Neural Cellular Automata (NCA)

NCA-generated data contains no human words at all. Instead, it evolves via a neural local rule:

s_{t+1}(x) = f_\theta(s_t(x), \mathcal{N}(x))

where $s_{t+1}(x)$ is the next state at position $$x$$ , $f_\theta$ is a neural evolution rule, and $\mathcal{N}(x)$ is the neighborhood of $$x$$ .

Though wordless, this data encodes temporal order and causal structure — like a child learning martial arts through movement before learning to read.

Reported Results (GSM8K)

| Training fuel | Tokens used | GSM8K result | Character | | :--- | :--- | :--- | :--- | | Real human text (C4) | 1,600,000,000 | Baseline | Heavy, expensive | | NCA synthetic data (2026) | 164,000,000 🚀 | +6% ⚡ | Lean, powerful |

The 10:1 efficiency ratio is attributed to the higher logical density of the synthetic data.

Attention as the Soul: Inductive Biases

Ablation studies reported in the paper show that resetting the attention weights erases the benefit, while modifying only MLP layers preserves it. NCA training, the author argues, sharpens attention's "eye" for structure — an inductive bias rather than memorized knowledge.

Conclusion

The post's takeaway: intelligence lies not merely in "knowing" but in "perceiving." Words are the clothing of intelligence; logic is its marrow. NCA pretraining suggests that stepping back from the scramble for human text — toward evolution's underlying structure — may open the door to stronger AI.

References

1. arXiv:2603.10055: *Training Language Models via Neural Cellular Automata* (2026). 2. Wolfram, S., *A New Kind of Science*. 3. Vaswani et al., *Attention Is All You Need*. 4. *The Efficiency of Structured Synthetic Data in Pre-pre-training*. 5. *Biological Scaling and the Limits of Language-centric AI*.

> Editor's note: The cited arXiv ID (2603.10055) and 2026 date could not be independently verified; readers should treat the quantitative claims as reported by the forum author rather than confirmed findings.