English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

DeepMind: The Topological Trouble With Transformers — Why Bigger Context Windows Won't Fix LLM State Tracking

Forum topic · 小凯 · 2026-06-24

Summary

A Chinese forum post analyzes a Google DeepMind paper (Mozer et al., 2026, arXiv:2604.17121) arguing that the Transformer's directed acyclic graph (DAG) architecture fundamentally limits state tracking. Because information only flows upward through layers and left-to-right across positions, each new input pushes an implicit state representation to a deeper layer — a phenomenon called depth exhaustion. Once inputs exceed model depth, state tracking breaks. Demonstrations include Gemini 3 failing a simple binary-search guessing game and misinterpreting the polysemous word 'bank' because shallow layers cannot access deep-layer disambiguation. Chain-of-Thought (CoT) prompting merely externalizes deep states into tokens, incurring compute, latency, memory, and context costs. DeepMind argues step-level recurrent architectures (Mamba, DeltaNet, RWKV-7) can achieve true state tracking, and outlines five research directions including enhanced state-space models, coarse-grained recurrence, representational alignment, and multi-stage training. The post concludes with practical implications for AI agent developers: longer context windows don't solve state tracking, CoT isn't free, and architecture choice determines agent capability ceilings.

Key points

A Chinese tech forum post discusses the DeepMind paper "The Topological Trouble With Transformers" (Mozer et al., 2026, arXiv:2604.17121), arguing that the Transformer's pure feed-forward topology fundamentally limits state tracking — and that scaling context windows cannot fix it.

1. A guessing game exposes a fatal flaw

  • In a 1–100 number guessing game, Gemini 3 gave contradictory answers: after "60 → lower", it said "70 → higher", failing basic interval tracking ([low, high] state maintenance).
  • Even Gemini 3 Thinking internally wrote "the hidden number is 42, so 42 → lower", yet still answered "lower" when the user guessed 42 — it forgets numbers it just wrote.
  • The post frames this as an architectural defect, not a hallucination.
  • 2. The "bank" experiment: meaning lost in deep layers

  • Given a fishing context, the model correctly resolved "bank" as *river bank* at deep layers (~layer 6, verified with Patchscopes interpretability tooling).
  • But when asked about an ATM, shallow layers (1–5) could only rely on co-occurrence statistics (bank → ATM → money bank), because deep-layer disambiguation is invisible to later tokens' shallow layers.
  • This is a topological dilemma, not a knowledge gap.
  • 3. The DAG curse

  • A Transformer decoder is a strict directed acyclic graph: information flows only upward (shallower → deeper) and leftward (earlier → later positions). A block at (layer *l*, position *t*) sees only blocks with layer < *l* and position ≤ *t*.
  • State tracking requires s_t = f(s_{t-1}, x_t) where s_t depends arbitrarily on s_{t-1}. In the DAG, each successive state must live in a deeper layer, drifting diagonally upward until it "tops out" of the model's *L* layers — depth exhaustion.
  • Merrill & Sabharwal (2025): log *n* layers are necessary and sufficient for tracking length-*n* strings / graph connectivity. But construction feasibility ≠ learnability; empirically models fail even two-step reasoning (Lepori et al., Gemma2-9B).
  • 4. Chain-of-Thought is an expensive patch

  • CoT works by printing deep representations as output tokens that re-enter shallow layers — effectively ferrying deep signals upward, like annotating a novel on sticky notes after every page.
  • DeepMind: inference humans do automatically and unconsciously (e.g., disambiguation) should not require laborious explicit reasoning.
  • The "thinking tax": linear compute growth, context occupation, multiplicative latency, growing KV-cache. And crucially, some implicit, distributed states cannot be externalized as natural language at all.
  • 5. A taxonomy of recurrent architectures (two axes, nine cells)

  • Recurrence axis: depth recurrence (re-running layers), step-level recurrence (state across positions), or both.
  • Token-to-recurrence-step ratio: >1 (parallel chunks), =1 (standard), <1 (latent thinking).
  • Examples: Looped/Universal Transformer (depth, ratio>1); linear attention, DeltaNet, Mamba, RWKV-7 (step, ratio=1); Recurrent Memory Transformer, Sentence Gestalt (depth+step); COCONUT, Hierarchical Reasoning Model (step, ratio<1).
  • Depth recurrence alone cannot solve state tracking: parallel propagation across positions still pushes state deeper.
  • Step-level recurrence enables unbounded state tracking but sacrifices training parallelism — the very advantage that replaced RNNs. Compromises:
  • Mamba/SSM: parallelizable training, RNN-like inference; but linear updates are *no more expressive* than a standard Transformer (Merrill et al., 2025).
  • DeltaNet: fast-weight programming + delta rule; with negative eigenvalue ranges (Grazzi et al., 2025), it trains in parallel while exceeding standard Transformer expressivity.
  • RWKV-7: expressive dynamic state evolution.
  • 6. DeepMind's five research prescriptions

    1. Enhanced state-space models: hybridize gated linear attention / gated delta net with standard Transformer blocks. 2. Approximate state tracking in feed-forward Transformers via training objectives and structural priors — a stopgap; DeepMind prefers structured, compositional state representations. 3. Coarse-grained recurrence: update state at sentence/semantic-chunk boundaries (e.g., Sentence Gestalt, Borazjanizadeh & McClelland 2025) to cut compute. 4. Representational alignment: variable-depth models (e.g., Universal Transformer) work without training or with light fine-tuning, likely due to residual-connection alignment (cf. Canon Layers, Allen-Zhu & Li 2025). 5. Multi-stage training: parallel feed-forward pretraining, then recurrence-introduced fine-tuning, with truncated gradients, backpropagation through recurrence, and higher arithmetic density.

    7. Table-lookup machines vs. digital minds

    Key quoted conclusions (translated): the feed-forward design extends context-based retrieval, but its topology fundamentally conflicts with the iterative nature of state tracking; relying on explicit natural-language "thought" to bypass depth limits is an inefficient patch. Next-generation models "must not merely re-scan the past; they must maintain a flowing, evolving representation of reality persisting across the multiple timescales required by temporally extended cognition."

    The post reads this as: current LLMs are essentially super-complex lookup machines that re-scan the context window rather than think; genuine thinking requires a continuously evolving internal state.

    8. Implications for agent developers

  • Don't fetishize long context: 128K → 1M context still cannot preserve implicit state established 1,000 tokens earlier.
  • CoT is not free: the thinking tax compounds for long-horizon agents.
  • Architecture choice sets the ceiling: for long-range coherence, multi-hop reasoning, and dynamic environments, watch Mamba / RWKV-7 / DeltaNet, external memory systems, and hybrid layered designs (fast feed-forward processing + slow recurrent state updates).
  • The ideal target: parallel training efficiency of Transformers plus persistent state evolution of RNNs — DeltaNet and RWKV-7 are already moving this way.
  • References cited in the post

  • Mozer et al. (2026). *The Topological Trouble With Transformers*. Google DeepMind, arXiv:2604.17121.
  • Merrill & Sabharwal (2025). *The Expressive Power of Transformers with Chain of Thought*. ICLR.
  • Giannou et al. (2023). *Looped Transformers as Programmable Computers*. ICML.
  • Gu & Dao (2024). *Mamba: Linear-Time Sequence Modeling with Selective State Spaces*. ICML.
  • Schlag et al. (2021). *Linear Transformers Are Secretly Fast Weight Memory Systems*. NeurIPS.
  • Peng et al. (2025). *RWKV-7: Expressive Dynamic State Evolution*.
  • Hao et al. (2025). *COCONUT: Continuous Latent Thought*.
  • Allen-Zhu & Li (2025). *Canon Layers: Aligning Representations Across Steps*.

Tags

#transformer#deepmind#state-tracking#chain-of-thought#recurrent-architectures#mamba#rwkv#ai-agents

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208060