Overview
A 41-page paper by Yan Zhou (School of Mathematics and Statistics, Changsha University of Science and Technology), posted to arXiv in May 2026 (arXiv:2605.05066), establishes an information-theoretic hard boundary—described as unavoidable as the speed of light—for all long-sequence models. Rather than attacking any specific architecture, the paper builds a unified mathematical framework that places Transformers, Mamba, linear attention, state-space models, and all their hybrids inside one impossibility triangle.
The Unified Abstraction: Online Sequence Processor (OSP)
The paper's central abstraction, the Online Sequence Processor (OSP), is a seven-tuple \((\mathcal{X}, \mathcal{S}, f, g, h, s_0, T)\) that formalizes all autoregressive sequence models:
- \(\mathcal{X}\): input space
- \(\mathcal{S}\): state space
- \(f\): state transition function
- \(g\): output function
- \(h\): update rule
- \(s_0\): initial state
- \(T\): sequence length
- Data Processing Inequality: information extracted after passing data through the compressed state \(s_t\) cannot exceed what is available in the original inputs, bounding retained information.
- Fano's Inequality: correctly recalling \(n\) facts with high probability requires sufficient mutual information; combined with the first, this strictly limits recallable facts under bounded state size.
- For continuous-state systems, the argument extends via Lipschitz stability.
- GLA, the best among tested, utilizes only ~0.04% of the theoretical limit.
- The gap stems from real-world constraints: the state must also encode positional information, padding tokens, and task structure, and practical encodings are far from information-theoretically optimal.
- Title: *The Impossibility Triangle of Long-Context Modeling*
- arXiv ID: 2605.05066
- Published: 2026-05-06
- Author: Yan Zhou, School of Mathematics and Statistics, Changsha University of Science and Technology
- Length: 41 pages, 6 figures
- Framework: OSP seven-tuple abstraction unifying Transformers, SSMs, linear recurrent networks, and hybrids
- Proof tools: Data Processing Inequality + Fano's Inequality; Lipschitz stability extension for continuous-state systems
- Quantitative theorem: models satisfying \(\mathfrak{E}\) and \(\mathfrak{C}\) recall at most \(\mathcal{O}(\text{poly}(d)/\log V)\) key-value pairs
- Classification: 52 architectures analyzed; Experiments: 5 representative architectures on associative recall; GLA best at ~0.04% bound utilization
This brings Transformers (KV-cache as state), state-space models (fixed-dimension hidden state), and linear recurrent networks (accumulated outer-product state) into a single formal system.
The Three Vertices of the Triangle
| Property | Symbol | Meaning | |:---------|:------:|:--------| | ⚡ Efficiency | \(\mathfrak{E}\) | Per-step compute is independent of sequence length | | 📦 Compactness | \(\mathfrak{C}\) | State size is independent of sequence length | | 🧠 Recall | \(\mathfrak{R}\) | Number of retrievable historical facts grows proportionally with sequence length |
Any model can occupy at most two of the three vertices. The core theorem:
In plain terms: if a model is both efficient and compact, the number of key-value pairs it can recall has a fixed upper bound depending only on model dimension \(d\) and vocabulary size \(V\)—completely independent of sequence length \(T\). Extending the context from 1K to 1M tokens adds no recall capacity.
Proof Tools
Key Findings
1. Classification of 52 architectures. Every long-sequence architecture published before March 2026 was mapped onto the triangle—none escapes its boundary:
| Architecture | Efficiency | Compactness | Recall | Strategy | |:-------------|:----------:|:-----------:|:------:|:---------| | 🔥 Transformer (standard) | ❌ depends on \(T\) | ❌ \(O(T)\) | ✅ full | R + E (sacrifices C) | | 🐍 Mamba / SSM | ✅ fixed | ✅ fixed | ❌ limited | E + C (sacrifices R) | | ⚡ Linear Transformer | ✅ fixed | ✅ fixed | ❌ limited | E + C (sacrifices R) | | 🌟 GLA | ✅ fixed | ✅ fixed | ❌ limited | E + C (sacrifices R) | | 🔀 Hybrid (Mamba+Attn) | partial | partial | partial | trade-off curve inside the triangle |
2. Experiments: reality is harsher than theory. Testing Transformer, Linear Transformer, Mamba (N=4,8,16,32,64), GLA, and a Hybrid on synthetic associative recall tasks showed all architectures fall strictly below the theoretical bound:
3. Hybrids cannot escape. The popular narrative that Mamba+Attention hybrids combine the best of both worlds is only partly true: by tuning the attention ratio \(r_{\text{attn}}\), hybrids trace a continuous trade-off curve inside the triangle, but the curve's endpoint always lies on a triangle edge—it never touches the third vertex:
Implications for the Industry
The paper directly challenges the long-context boom of the past 18 months. Mamba-style architectures were marketed as Transformer successors on the promise of "linear complexity + fixed state"—i.e., E + C—but the recall cost was rarely questioned. Claiming 1M-token context does not mean recalling 1M tokens of content. Scaling a state-space model to 100B parameters or a 10K state dimension does not escape the \(\mathcal{O}(\text{poly}(d)/\log V)\) cage.
Like the CAP theorem for distributed databases, the impossibility triangle does not prevent building useful systems—but it demands honest choices. As the paper concludes: every long-sequence architecture is a choice about which vertex to approach and which property to sacrifice. The long-context race has changed tracks: from "who can do all three" to "who can do two best, while admitting the third is out of reach."