The Twilight of Transformers: Memory Cache and CTM Challenge the Quadratic Complexity Curse

Since its birth in 2017, the Transformer has dominated deep learning for eight years. BERT, GPT, T5, LLaMA, Claude — nearly every famous large model is built on the same core: the attention mechanism, which lets the model directly "see" relationships between any two tokens regardless of distance. The price is brutal: computational complexity grows with the square of sequence length. As context scales from 4K to 128K to 1M, costs explode rather than grow linearly.

In 2025–2026, two research lines from very different directions point at the same question: can we keep Transformer-level capability while escaping the quadratic complexity curse?

Google Research's Memory Caching team found an answer in the old RNN. Llion Jones of Sakana AI — one of the eight authors of the Transformer paper — took a more radical path: bringing the dimension of time back into AI architecture.

Key points

Why Transformers are hitting a ceiling

• Quadratic complexity: self-attention computes an L×L matrix for sequence length L — O(L²) time and space, the root cause of quadratic KV cache growth. Processing 1M tokens requires ~1 trillion attention operations versus ~1M for an RNN of the same size — a million-fold gap.
• Long-context amnesia: beyond a threshold, models suffer "Context Rot" (Chroma, 2025) — they "remember" but cannot efficiently retrieve, like losing a key in a cluttered warehouse.
• Cost structure: roughly 80% of Transformer inference cost goes to attention, not parameters. Shrinking the model doesn't fix long-sequence costs — a structural tax on customer service, legal documents, codebases, and multi-turn dialogue.

Memory Caching: Google's way to revive the RNN

The key insight (Ali Behrouz et al., Google Research, arXiv:2602.24281): an RNN's fixed memory is an implementation choice, not a theoretical limit. If the RNN can save key memory segments and retrieve them later — like the brain converting short-term to long-term memory — memory capacity grows with the sequence.

The mechanism is simple:

1. Segmentation: split the sequence into fixed-length segments (e.g., 256 tokens). 2. Online memory: a standard RNN processes each segment with a fixed-size state vector. 3. Cached memory: each segment's final state is saved to a read-only cache pool accessible by later tokens. 4. Aggregation: when processing segment s, the model aggregates all previously cached memories alongside online memory.

Four aggregation strategies, from simple to complex:

| Strategy | Mechanism | Notes | |---|---|---| | Residual Memory | simple sum of current + all cached memories | basic; linear memory mathematically collapses | | Gated Residual Memory (GRM) | context-aware gates γ dynamically weight each segment's contribution | recommended; strong results | | Memory Soup | parameter-level weighted averaging | suits deep memory modules (DLA, Titans) | | Sparse Selective Caching (SSC) | MoE-style router with Top-k cache selection | most inference-efficient |

GRM formula:

\[y_t = \gamma_t^{(s)} * M_t^{(s)}(q_t) + \sum \gamma_t^{(i)} * M_L^{(i)}(q_t)\]

where γ depends on both the current token and historical segment content — the model selectively reads relevant history instead of blindly mixing everything.

Theoretical unification: when segment size = 1 with value-less vector memory, Memory Caching is mathematically equivalent to gated global attention. Transformers are an extreme special case, yielding a continuum:

• N=1 → O(L²), equivalent to a Transformer
• N=L → O(L), pure RNN
• N=L/C (constant segmentation) → O(L²/C), practical compromise
• N=log L → O(L log L)

Experimental results (760M parameters, 30B tokens):

• Language modeling: Titans+GRM beats Transformer++ (Wiki ppl 19.14 vs 24.18; average accuracy 52.55% vs 49.64%, +0.99%).
• Needle-in-a-Haystack at 16K context, hardest UUID task (S-NIAH-3): baseline DLA only 4.0%; DLA+GRM 18.2% (4.5×); Titans+GRM 32.2% (near Transformer's 40.8%).
• Contextual retrieval (SWDE, SQuAD, FDA): Titans+GRM averages 40.50, close to Transformer's 41.00 (baseline Titans only 31.75).

CTM: Sakana AI brings time back

Llion Jones argues that LLMs show "jagged intelligence" — brilliant on some tasks, failing at trivial ones — because the architecture ignores time. Most networks, including Transformers, produce output almost immediately after input; Chain-of-Thought simulates gradual thinking only through full forward passes.

The Continuous Thought Machine (Darlow, Jones et al., Sakana AI, NeurIPS 2025) makes time an intrinsic dimension: an internal time axis from tick 1 to T, decoupled from input tokens. Four components:

1. Pre-trained encoder converting raw input to features. 2. Synapse: a U-Net producing pre-activations from features and the previous tick's states. 3. Neuron-level models: each neuron has its own small MLP processing a history window (M = 10–100 ticks) — per-neuron memory and temporal dynamics. 4. Neural synchronization: a matrix S_t = Z_t × Z_t^T of temporal synchronization between neurons, sampled as the latent representation for attention and output.

Adaptive compute: the model decides when to stop based on its own output entropy — trained via losses at the lowest-loss tick and the highest-confidence tick. Easy inputs finish fast; hard inputs get more thinking time.

Experiments: trained on 39×39 mazes (paths up to 100 steps), CTM generalizes to larger 99×99 mazes and longer paths, while LSTM baselines fail entirely. It exhibits "episodic future thinking" — imagining future maze states without positional encodings — an emergent planning capability. Ablations show neuron-level models and synchronization are both required: removing either drops performance from 66% to below 50%.

Comparing the two paths

| Dimension | Memory Caching | CTM | |---|---|---| | Motivation | Transformer efficiency | missing time in AI | | Math basis | RNN + segmented caching + gated aggregation | neural dynamics + synchronization + internal time | | Complexity | O(NL), interpolable to O(L²) or O(L) | RNN-like, variable internal ticks | | Relation to Transformer | unifying framework (Transformer is a special case) | outright replacement | | Inspiration | computer science (caching, routing) | neuroscience (oscillations, synchronization) | | Engineering maturity | high (integrates existing RNNs) | medium (new training pipeline) | | Scale validation | 760M params, 30B tokens | small scale (~9M-param maze) |

• For replacing Transformers today, Memory Caching is pragmatic: it boosts existing architectures (Mamba, Titans, DLA), is validated at scale, offers tunable complexity, and fits existing training infrastructure.
• For making AI more brain-like, CTM is more radical: it targets cognitive architecture rather than engineering optimization, and its emergent planning hints at the "generative cognition" direction Rich Sutton describes.

Implications for Scaling Laws and beyond

Memory Caching recalibrates compute-efficiency scaling (same compute handles longer sequences); CTM redefines inference-time scaling (dynamic allocation of thinking time). For future world models requiring continuous perception, long-term memory, dynamic planning, and causal reasoning, attention naturally handles the first two but struggles with the last two — it has no persistent state and no temporal dynamics.

Outlook

A plausible fusion: CTM's neuron-level models could process per-segment temporal dynamics while Memory Caching's GRM gates make latent selection more principled. Timeline: Memory Caching could see production deployment by 2026; CTM likely needs 3–5 years of validation at LLM scale.

Conclusion

Transformers won't disappear tomorrow — they remain the strongest, most mature architecture. But these papers mark the formal start of the post-Transformer era. Choose Memory Caching if you care about efficiency, cost, and deployability; choose CTM if you believe time and neural dynamics are key to machine intelligence. Most likely, both will coexist: Memory Caching as the mainstream near-term alternative, CTM as a long-term research bet.

References

1. Memory Caching: RNNs with Growing Memory (2026.02) — Ali Behrouz et al., Google Research. arXiv:2602.24281 2. Continuous Thought Machines (2025) — Luke Darlow, Llion Jones et al., Sakana AI. NeurIPS 2025 3. JetBrains Research (2025.12) — Smarter Context Management for LLM-Powered Agents, NeurIPS 2025 4. Chroma (2025) — Context Rot in Long-Context Retrieval 5. Rich Sutton (2025) — The Generative Turn in Cognitive Architecture