AI Paradigm Shift: From the Transformer Dead End to the CTM Era
*An English adaptation of a Chinese-language analysis originally published on zhichai.net.*
Key points
- The inventor's paradox. Llion Jones, one of the eight co-authors of "Attention Is All You Need" (2017) and the person who named the Transformer, declared in 2024–2025 that he is "absolutely sick" of the architecture and has "significantly reduced" his Transformer research time since early 2024. The original paper has been cited over 100,000 times.
- "Dead end" warning at the industry's peak. Despite massive global AI investment (reported at over $150 billion), OpenAI's near-$100B valuation, and GPT-4's expert-level performance, Jones warns that AI has "calcified" around a single architectural approach, blinding researchers to the next breakthrough. He cites a severe "exploitation vs. exploration" imbalance.
- "Jagged intelligence" as evidence. GPT-4 exhibits genius-level behavior (academic writing, complex code generation, domain problem-solving) alongside "idiot" failures (multi-step arithmetic, basic logic puzzles), exposing a lack of genuine compositional reasoning and understanding.
- Static, one-pass computation. The standard Transformer is described as a "massive, pre-calculated mathematical function" whose reasoning depth is strictly bounded by its layer count. It cannot pause, reflect, backtrack, or allocate more computation to harder problems.
- Pattern matching, not thinking. As one cited analysis puts it: Transformers "don't actually 'think'. They match patterns."
- Three structural deficits: 1. No planning capability — cannot decompose complex goals into subgoal sequences, causing multi-step reasoning failures. 2. No consistency checking — cannot detect logical contradictions in its own outputs. 3. No causal understanding — conflates correlation with causation and cannot perform counterfactual reasoning.
- The Scaling Law double-edged sword. Predictable power-law gains have de-risked investment and enabled precise resource planning, but per Jones, "scaling has sucked all the oxygen out of the room," marginalizing architectural innovation and locking talent into the incumbent paradigm.
- Introduces internal temporal dynamics, letting the model iterate and refine internal states over time rather than produce output in one forward pass.
- Enables adaptive computation depth: easy problems get less "thinking," hard problems get more — mirroring biological cognition.
- Supports genuine reasoning behaviors (reflection, backtracking) that static architectures structurally cannot express.
- https://venturebeat.com/technology/sakana-ais-cto-says-hes-absolutely-sick-of-transformers-the-tech-that-powers
- https://algustionesa.com/why-a-transformer-co-creator-is-sick-of-his-own-ai/
- https://eu.36kr.com/en/p/3643193251516297
- https://www.linkedin.com/pulse/beyond-transformer-why-ai-needs-time-think-murat-durmus-oipve
Why the Transformer hits a wall
CTM: a brain-inspired paradigm leap
The article presents the Continuous Thought Machine (CTM) — associated with Sakana AI, where Llion Jones serves as CTO — as the proposed alternative:
Transformer vs. CTM: structural comparison
| Dimension | Transformer | CTM | |---|---|---| | Computation | Fixed-depth, one pass | Dynamic, iterative over internal timesteps | | Reasoning depth | Bounded by layer count | Adaptive to task difficulty | | Uncertainty handling | Single guess | Iterative refinement, self-correction | | Training | Mature, hardware-friendly | Early stage, research frontier |
Outlook
The article argues the field must rebalance exploitation (scaling) and exploration (new architectures). Whether CTM-class models can match Transformer efficiency at scale remains an open question, but the timing of a co-creator's defection marks a notable inflection point in the AGI debate.