The Topological Trouble With Transformers: Why Longer Context Windows Won't Save LLMs

Key points

• The architectural flaw: A Transformer block at layer *l* and position *t* can only see blocks at strictly lower layers and earlier positions. It is a pure directed acyclic graph (DAG) with no cycles or recurrence.
• State tracking demands recurrence: Because s_t = f(s_{t-1}, x_t), the representation of s_t must live deeper than s_{t-1}. As tokens are processed, the state drifts diagonally upward and is "pushed out the top" of the network once the sequence length exceeds the depth (depth exhaustion).
• Empirical evidence from DeepMind:
• *Guess-the-number*: Gemini 3 cannot maintain a [low, high] interval, even though its hidden reasoning names the correct hidden number.
• *Word-sense disambiguation*: Tools like Patchscopes show layers 1–5 hold an ambiguous "bank" embedding, layer ~6 correctly resolves to *river bank*, but later shallow tokens cannot access that disambiguation and revert to the money-bank prior when "ATM" appears.
• Chain-of-thought is an expensive patch: CoT externalizes deep representations as output tokens so they re-enter as shallow tokens. Costs scale linearly with thought length in compute, context, latency, and KV-cache memory, and many internal states cannot be verbalized at all.
• A taxonomy of recurrent Transformers (two axes × three ratios):
• *Depth recurrence* (Looped Transformer, Universal Transformer, RINS): adds expressivity but still suffers from depth exhaustion across positions.
• *Step recurrence* with ratio = 1 (Block-recurrent Transformers, Linear Attention, DeltaNet, Mamba, RWKV-7): enables true unbounded state tracking.
• *Step + depth* with ratio < 1 (COCONUT, Hierarchical Reasoning Model, CYB): "latent thinking" with multiple recurrence steps per token.
• Promising architectures:
• Mamba / SSM: parallel training (convolution-like), RNN-style inference, but linear updates do not exceed standard Transformer expressivity (Merrill et al., 2025).
• DeltaNet: fast-weight programming with the delta rule; extends to negative eigenvalues (Grazzi et al., 2025) to gain expressivity over vanilla Transformers while staying parallelizable.
• RWKV-7: expressive dynamic state evolution, listed as capable of real state tracking.
• Five research directions proposed by DeepMind:
1. Hybrid SSM + Transformer blocks (e.g., Gated Linear Attention, Gated Delta Net). 2. Train feedforward Transformers with objectives and structural priors that approximate state tracking. 3. Coarse-grained recurrence at sentence or semantic-chunk level (e.g., Sentence Gestalt). 4. Exploit representational alignment from residual streams; Canon Layers align representations across input positions. 5. Multi-stage training: parallel feedforward pretraining, then fine-tune with recurrence (truncated gradients, recurrent backprop, increased arithmetic density).
• Implications for AI agents: Longer context windows do not fix the issue; CoT taxes compound over long horizons; agents needing long-horizon consistency, multi-hop reasoning, or dynamic-environment interaction should track recurrent architectures (Mamba, RWKV-7, DeltaNet), external memory (e.g., MemoryWAM), and hierarchical fast/slow designs.

References

• Mozer et al. (2026). *The Topological Trouble With Transformers*. Google DeepMind, arXiv:2604.17121.
• Merrill & Sabharwal (2025). *The Expressive Power of Transformers with Chain of Thought*. ICLR.
• Giannou et al. (2023). *Looped Transformers as Programmable Computers*. ICML.
• Gu & Dao (2024). *Mamba: Linear-Time Sequence Modeling with Selective State Spaces*. ICML.
• Schlag et al. (2021). *Linear Transformers Are Secretly Fast Weight Memory Systems*. NeurIPS.
• Peng et al. (2025). *RWKV-7: Expressive Dynamic State Evolution*.
• Hao et al. (2025). *COCONUT: Continuous Latent Thought*.
• Allen-Zhu & Li (2025). *Canon Layers: Aligning Representations Across Steps*.