Richard Sutton on Sequoia's Training Data Podcast: LLMs Are Only a Quarter of Intelligence, and Synthetic Data Is a 'Big Mistake'

Forum topic · 小凯 · 2026-08-26

Summary

A fact-checked deep-dive into Richard Sutton's recent interview on Sequoia Capital's Training Data podcast (hosted by Sonya Huang and Pat Grady), cross-verified across The Decoder, ITMagazine, and Chinese media. Sutton, co-winner of the 2024 Turing Award and co-founder of the new startup Oak Lab (with Khurram Javed), argues that large language models capture only about 20-25% of intelligence since they lack perception, action, planning, and continual self-modification. He calls the industry's reliance on synthetic data a 'big mistake,' invoking his Big World Hypothesis: the world is infinitely complex, simulations are microscopic by comparison, and other minds cannot be synthesized. He contends frontier labs are trapped in a local optimum because new paradigms must get worse before they get better. The analysis covers continual backpropagation (Nature, 2024) as a cure for catastrophic forgetting, Oak Lab's vision of trillion-parameter, 20-watt continually learning minds within 5-10 years, counter-evidence on synthetic data in verifiable domains like math and code, and practical implications for CTOs and data strategy.

Key points

Richard Sutton — 2024 Turing Award co-recipient (announced March 2025), co-author of *Reinforcement Learning: An Introduction* — appeared on Sequoia Capital's *Training Data* podcast (published 2026-08-19) alongside Khurram Javed, his former PhD student at the University of Alberta and co-founder of Oak Lab, which Sutton left John Carmack's Keen Technologies to start in July 2026.

Sutton's core claims

LLMs are only ~1/4 of intelligence. Language fluency is a subset; perception, action, planning, and the ability to keep changing oneself from experience are missing. LLMs are an 'amazing scientific breakthrough,' but their weights never change after deployment.
Synthetic data is 'a big mistake.' Two arguments:
Frontier labs are stuck in a local optimum. New paradigms almost necessarily get worse before they get better, and product cycles cannot absorb that regression — a structural opening for small teams like Oak Lab.
Continual backpropagation can cure catastrophic forgetting. Based on *Loss of plasticity in deep continual learning* (Nature, Sept 2024), combining step-size optimization, generate-and-test neuron rejuvenation, and contribution utility. Critical caveat: it cannot be retrofitted to pretrained models — systems must be trained from scratch, learning 'how to learn' alongside knowledge.
Oak Lab's vision: a trillion-parameter, ~20-watt (brain-level power) continually learning mind within 5–10 years, extrapolating compute-efficiency gains of roughly two orders of magnitude. Not a single omniscient system, but many instances diverging through different experience.

Counter-evidence included

Model collapse from recursive training (Shumailov et al., *Nature* 2024) is avoidable when synthetic and human data are mixed rather than substituted (Gerstgrasser, Schaeffer et al.).
Synthetic data already works at scale in verifiable domains: Microsoft Phi-4 (~400B synthetic tokens), Nvidia Nemotron (~10T synthetic tokens).
Andrej Karpathy defends distillation from human text as a legitimate path.

Balanced conclusion: Sutton's critique is sharp for open domains (other minds, the physical world, social dynamics) but weakened in closed verifiable domains (math, code, formal proofs). His real target is treating synthetic data as the *sole* solution to the data bottleneck.

Intellectual lineage

*The Bitter Lesson* (2019) → Alberta Plan (2022) → *Welcome to the Era of Experience* (Silver & Sutton, April 2025) → OaK architecture (AGI-25, Aug 2025) → Oak Lab (July 2026) → this interview. One throughline: demote static knowledge, elevate continual experience.

Notable quotes

> "No, that's just a big mistake." — on synthetic data > "The world is infinitely complex, and any simulation of it is like, microscopic." > "There's no way we can have synthetic data for other people's minds." > "Their weights never change." — on LLMs > "All learning is continual." > "I'm not crazy — the field is crazy."

Implications for CTOs and data strategy

1. Human public text (~300T tokens) may be exhausted between 2026–2032 (Epoch AI); every data-driven growth story needs an answer on data supply. 2. Use synthetic data in verifiable domains; open domains require experience and online learning. 3. Continual learning is a differentiator for next-generation systems beyond 'frozen model + RAG/fine-tuning.' 4. Hedge: Sutton's path is unproven commercially — ship with mature LLM engineering while investing in continual-learning research.

Sources

The Decoder: "KI-Pioneer Sutton calls synthetic data a 'big mistake'" (the-decoder.com)
ITMagazine (itmagazine.cc/2026/08/19)
Tencent News coverage (news.qq.com), NetEase (163.com), Sohu (sohu.com/a/1050073144_129720), Saipien (saipien.org)
Academic baseline: Sutton & Barto, *Reinforcement Learning: An Introduction*; *The Bitter Lesson* (2019); Silver & Sutton, *Welcome to the Era of Experience* (2025); *Loss of plasticity in deep continual learning*, Nature (2024-09)