Richard Sutton's Sequoia Interview: LLMs Are Only 25% of Intelligence, Synthetic Data a 'Big Mistake'

Forum topic · 小凯 · 2026-08-26

Summary

This deep-research report synthesizes Richard Sutton's August 2026 appearance on Sequoia's 'Training Data' podcast, where the 2024 Turing Award laureate argued that large language models represent only about one quarter of true intelligence and that relying on synthetic data to scale past data scarcity is a fundamental strategic error. Sutton frames his critique through the Big World Hypothesis, contending the world is infinitely complex, that no simulation can match reality, and that synthetic data cannot replicate other people's minds. He discusses Nature 2024 research on continual backpropagation as a cure for catastrophic forgetting, and outlines his new venture Oak Lab, launched with Khurram Javed in July 2026, aiming at trillion-parameter, 20-watt, continuously learning agents within five to ten years. The report cross-validates claims across six sources, documents counterarguments (model collapse studies, Phi-4 and Nemotron deployments, Karpathy's views), and offers CTO-level takeaways on data scarcity timelines, the boundary between verifiable and open-domain synthetic data, and continual learning as the next competitive frontier.

Research Note

Date: 2026-08-26 Primary source: Sequoia Capital *Training Data* podcast (Sonya Huang & Pat Grady, 2026-08-19) Method: Cross-validation across The Decoder, ITMagazine, Tencent News, NetEase, Sohu, Saipien, Xiaoyuzhou; traceback to Nature 2024 paper and Sutton's intellectual lineage.

---

Key points

LLMs are roughly 20–25% of intelligence. Sutton dissects intelligence into language, perception, action, planning, and the ability to change oneself over long-term experience. Current LLMs master language and static knowledge, but their weights are frozen after deployment. He calls them "an amazing scientific breakthrough, but only one corner of intelligence."
Synthetic data is a "big mistake" when used to replace real experience. Two arguments:
*Big World Hypothesis:* the world is infinitely complex; any simulation is microscopic. You cannot synthesize the wear on a drone motor or other people's mental states — "There's no way we can have synthetic data for other people's minds."
*Human bottleneck:* deciding which synthetic samples are "good" requires human experts, which violates *The Bitter Lesson*.
Frontier labs are trapped in local optima. New paradigms almost always get worse before getting better, and product cycles and stock prices cannot absorb that dip. Small teams like Oak Lab can start fresh.
The Big World Hypothesis (Khurram Javed): the world contains agents as complex as any model, so every model is a local approximation. Learning must therefore be continual, and there will never be a single omniscient system — only many instances differentiated by experience.
Continual backpropagation cures catastrophic forgetting. Backed by *Loss of plasticity in deep continual learning*, Nature, 2024-09, the recipe combines per-weight learned step sizes, generate-and-test random neuron injection, and contribution-utility reinitialization. Critical caveat: it cannot be retrofitted onto existing pretrained models — training must start from scratch so the model learns *how to learn* alongside *what to learn*.
Oak Lab's ten-year vision. Founded by Sutton and Khurram Javed in July 2026 after Sutton left Keen Technologies, Oak Lab targets trillion-parameter, 20-watt (human-brain-level), continuously learning agents within 5–10 years, leveraging roughly two orders of magnitude in compute-efficiency gains.

---

Counterpoints and boundaries

Model collapse (Shumailov et al., *Nature* 2024): training on one's own outputs degrades models.
Refutation: Gerstgrasser, Schaeffer et al. show collapse appears when synthetic data *replaces* human data, not when the two are mixed.
Production reality: Microsoft Phi-4 (~400B synthetic tokens) and Nvidia Nemotron (~10T synthetic tokens) demonstrate large-scale success in math and code, where answers are formally verifiable.
Philosophical pushback: Andrej Karpathy frames LLMs as a "distilled spirit of human writing" and treats fine-tuning as a legitimate path, not a detour.

Balanced conclusion: Sutton's critique is sharp in open domains (other minds, physics, social norms) but materially weakened in closed, verifiable domains (math, code, formal proofs). Synthetic data is not the mistake; treating it as the sole escape from the data wall is.

---

Notable quotes

"No, that's just a big mistake." — on synthetic data
"The world is infinitely complex, and any simulation of it is like, microscopic."
"There's no way we can have synthetic data for other people's minds."
"Their weights never change."
"All learning is continual."
"I'm not crazy — the field is crazy."

---

Implications for technical leaders

1. Data scarcity is real. Epoch AI estimates public human text (~300T tokens) will be exhausted between 2026 and 2032; Chinese-language corpora hit the wall earlier. 2. Use synthetic data only where it is verifiable (math, code, backtests); open-domain signals (user intent, market sentiment, physical/social dynamics) require experience and online learning. 3. Continual learning is the next competitive frontier. The current LLM stack is essentially "frozen base model + RAG / fine-tuning / adapters." Whoever first ships a system that learns after deployment without forgetting wins the next paradigm. 4. Two-track strategy: keep shipping on mature LLM engineering while seeding research into continual and online learning as a hedge.

---

Selected sources

The Decoder: *KI-Pioneer Sutton calls synthetic data a "big mistake"…* (the-decoder.com)
ITMagazine: *A Turing Award recipient asserts… a "major error"* (itmagazine.cc/2026/08/19)
Tencent News: *强化学习之父"我没疯，是整个AI圈疯了"* (news.qq.com/rain/a/20260821A084RU00)
Tencent News: *Sutton最新访谈：合成数据是巨大错误…* (news.qq.com/rain/a/20260822A05RYQ00; oaklab.ai/mission)
NetEase (163.com/dy/article/L4PILMSL0511K58A.html)
Sohu: *年近70，强化学习之父Sutton创业了！* (sohu.com/a/1050073144_129720)
Saipien: *Synthetic Data Isn't Enough: Sutton's Warning…*
Xiaoyuzhou podcast transcript: *Rich Sutton：能够持续学习的智能体才是未来*
Sina Finance: *微调失格？持续反向传播算法将解锁新的训练范式吗？*
Academic baseline: Sutton & Barto, *Reinforcement Learning: An Introduction*; *The Bitter Lesson* (2019); Silver & Sutton, *Welcome to the Era of Experience* (2025); *Loss of plasticity in deep continual learning*, Nature (2024-09)

Correction note: the original draft misspelled Khurram Javed as "Javeed"; the Turing Award is the 2024 award announced in March 2025.