Transformer Co-Creator Lukasz Kaiser: "Next-Token Prediction" Is a Dead Paradigm
> Person: Lukasz Kaiser, co-author of the Transformer paper, senior research scientist at OpenAI > Source: Multiple public interviews, October–November 2025 (Jon Hernandez, Matt Turck, OpenAI Forum, etc.) > Core claim: The scaling-law era of "predicting the next token" is near its ceiling; reasoning models are the next chapter of AI
This post is a Chinese forum summary of Kaiser's recent interviews, translated here into English.
1. From Logician to Transformer Architect
Kaiser's background is unusual for an AI researcher. He studied mathematics at the University of Wroclaw, earned a PhD in logic and automata theory at RWTH Aachen, and held a tenured research position in logic at Paris Diderot University. He only moved into deep learning when he joined Google Brain in 2013.
Four years later came "Attention Is All You Need." Of the eight authors — the so-called "Transformer Eight" — Noam Shazeer founded Character.AI, Aidan Gomez founded Cohere, Ashish Vaswani co-founded Essential AI, and Kaiser stayed on, joining OpenAI in 2021 and contributing to GPT-4, o1, o3, and GPT-5.
His conclusion now: the "predict the next token" path has run out.
2. Why Next-Token Prediction Isn't Enough
In October 2021, OpenAI published "Training Verifiers to Solve Math Word Problems." The paper made a stark calculation: solving high-school-level math word problems through pure scaling — more parameters, data, and compute — would require hundreds of billions of parameters or more, far beyond what GPT-3 (175B) could achieve.
Kaiser revisited this in his interviews. His conclusion: pure autoregressive next-token prediction mathematically cannot solve problems requiring multi-step reasoning.
Next-token prediction is essentially memory and pattern matching. Trained on internet text, models learn linguistic structure, factual associations, and stylistic imitation — enough to write, translate, and chat. But they don't "think." On problems requiring multi-step derivation, they either guess or fall into loops.
In Kaiser's words:
> Early LLMs mainly learned statistical associations, with no explicit intermediate reasoning steps. Reasoning models are trained to generate intermediate steps while solving problems — commonly called chains of thought.
3. What Makes Reasoning Models Different
OpenAI released the o1 series in September 2024, which Kaiser calls AI's "new paradigm." Its training differs fundamentally from prior LLMs:
1. Very little training data. Kaiser stresses that reasoning models use a "tiny amount of data" compared to internet-scale pretraining — relying instead on reinforcement learning on verifiable tasks. 2. Chain of thought is the core mechanism, not decoration. o1 generates internal intermediate reasoning steps (hidden chain of thought) before answering — like scratch paper for humans, letting the model spend more computation thinking. 3. Learning from less data, generalizing better. High-school math problems that would have required hundreds of billions of parameters in 2021 can now be solved by some of the smallest reasoning models — not because models got bigger, but because the training paradigm changed.
4. No AI Winter
Kaiser directly rejects "AI winter" narratives, for three reasons:
- The reasoning paradigm is still early — like Transformer in 2017, before anyone could foresee GPT-4 or ChatGPT.
- Low-hanging fruit remains. OpenAI has many optimizations skipped for prioritization reasons; engineering alone can keep improving performance.
- AI progress will resemble Moore's Law, extending its runway by exhausting each S-curve and finding the next one.
- GPT-5 is natively multimodal, trained on images and audio, able to generate both; video is coming.
- GPT-5 Pro runs multiple chains of thought in parallel, integrating them into a final answer — explaining its higher cost.
- Retraining is mainly about cost. GPT-4o is not stronger than GPT-4, just much cheaper.
- Pretraining persists as one step in the pipeline; post-training, reinforcement learning, and data filtering now contribute more to capability.
- Cogniscendo: "2025-10-29: Some confirmations from OpenAI" (Jon Hernandez interview breakdown)
- The MAD Podcast with Matt Turck: "OpenAI's Lukasz Kaiser" (November 2025)
- OpenAI Forum: "Learning Powerful Models: From Transformers to Reasoners and Beyond" (October 2025)
- 2025 Global Machine Learning Technology Summit keynote topic
- "Training Verifiers to Solve Math Word Problems" (Cobbe et al., 2021)
5. GPT-5 and Future Models
6. A Counterintuitive View: Video Data Isn't That Important
Despite the multimodal hype, Kaiser is skeptical of video data's value:
> We live in the physical world — that's what video contains. But in our heads there are many different worlds, expressed in text. Language models already possess a model of our abstract world.
Outside robotics and physical simulation, he argues, video volume contributes little to intelligence; abstract reasoning, logic, and human knowledge in text matter more. "World models" built from video won't generalize well, because the world in our minds is mostly abstract, not physical.
7. Competition, Ads, and AI Values
Competition is not a major pressure. People move between labs constantly; no secret stays buried long. Labs remain academically friendly.
No ad-driven design. Kaiser states:
> There's a strong culture inside OpenAI — among employees and parts of leadership — not to maximize engagement for advertising.
ChatGPT can now support shopping, but Kaiser insists recommendations cannot influence model outputs: "It's written very explicitly in the agreements. Biasing a language model via post-training toward certain things can produce very strange results."
More worried about AI weapons than AI slop.
> Any research result will be used in different ways; you can't control how. AI is very powerful, and we must accept it will be used in ways we don't want.
8. Kaiser's Research Direction: Learning Reasoning from Arbitrary Data
Kaiser's current focus is learning reasoning from arbitrary data — not just verifiable tasks like math and coding, but the fuzzy, subjective, real-world decisions that make up most of human reasoning. He sees this as the next major breakthrough.
9. Summary: The Future Through the Transformer Creator's Eyes
1. The paradigm shift has happened. Pretraining scaling laws have hit a ceiling; reasoning models represent models that "think" rather than just bigger models. 2. Data isn't the bottleneck — reasoning is. RL + chain of thought + verifiable tasks matter more than data volume. 3. Multimodality is necessary but not sufficient. True intelligence comes from the abstract worlds encoded in text. 4. No AI winter. The reasoning paradigm is early; engineering headroom is large; new S-curves will emerge. 5. Values matter. OpenAI avoids engagement-maximizing ad design; Kaiser worries more about AI weapons than AI-generated junk.
10. The Bigger Picture
Kaiser has witnessed a full paradigm cycle: Transformer in 2017 solved efficient sequence processing; o1 in 2024 solved how to make models think. The gap between these is larger than the one from RNNs to Transformers.
His 2025 Global Machine Learning Technology Summit talk is titled "Reasoning Models: History, Present, and Future" — reasoning models already have a history (the 2021 verifier paper), a present (o3, GPT-5), and a future (learning reasoning from arbitrary data).
> The dream is that at some point there will be a model that learns to be a great programmer, a great conversational agent, capable of both visual and language processing.