Twilight of the Transformer: Memory Cache and CTM Challenge the Quadratic Complexity Curse

Forum topic · 小凯 · 2026-06-07

Summary

This forum post examines two 2025-2026 research lines that challenge the Transformer's O(L²) attention complexity: Google Research's Memory Caching and Sakana AI's Continuous Thought Machine (CTM). Memory Caching (Ali Behrouz et al., arXiv:2602.24281) revives RNNs by segmenting sequences into cached memory blocks with four aggregation strategies—Residual, Gated Residual (GRM), Memory Soup, and Sparse Selective Caching. Notably, with segment size 1, Memory Caching mathematically reduces to global attention, unifying RNNs and Transformers on a complexity continuum. Experiments at 760M parameters show Titans+GRM beating Transformer++ on language modeling and lifting needle-in-a-haystack retrieval from 4% to 32.2%. CTM (Luke Darlow, Llion Jones et al., NeurIPS 2025) makes time a first-class dimension via neuron-level models and neural synchronization, with adaptive computation that stops when confident, and demonstrates emergent path-planning in mazes generalizing to unseen sizes. The post compares the two paths—pragmatic efficiency versus cognitive architecture—and argues a post-Transformer era has begun.

Since its birth in 2017, the Transformer has dominated deep learning for eight years. BERT, GPT, T5, LLaMA, Claude—nearly every major model is built on attention. Its strength is letting the model directly "see" relations between any two tokens; its cost is brutal: compute grows quadratically with sequence length, which becomes explosive as contexts scale from 4K to 128K to 1M tokens.

In 2025-2026, two research lines from very different directions target the same question: can we keep Transformer-level capability without the quadratic complexity curse?

Why the Transformer is hitting a ceiling

Quadratic complexity: self-attention produces an L×L matrix—O(L²) time and space. A 1M-token context implies ~1 trillion attention computations, versus ~1M for an RNN: a million-fold gap. This is an architectural flaw, not an optimization problem.
"Context Rot": Chroma's 2025 research shows retrieval efficiency declines with sequence length. Models "remember" but cannot "find"—like a key lost in a cluttered warehouse.
Structural cost inequality: ~80% of Transformer inference cost goes to attention, not parameters. Shrinking the model doesn't help long-sequence workloads (support, legal docs, codebases, multi-turn chat).

Google's Memory Caching: reviving the RNN

Google Research's Ali Behrouz team observes that an RNN's "fixed memory" is an implementation choice, not a theoretical limit. Memory Caching works in four steps:

1. Segmentation: split the sequence into fixed-size segments (e.g., 256 tokens). 2. Online memory: a standard RNN maintains a fixed-size state within each segment. 3. Cached memory: each segment's final state is stored read-only in a cache pool. 4. Aggregation: when processing a token, the model aggregates all previously cached memories.

Four aggregation strategies are proposed:

| Strategy | Mechanism | Notes | |---|---|---| | Residual Memory | simple sum of current + cached memories | basic, but linear memory mathematically collapses | | Gated Residual (GRM) | context-aware gate γ modulates each segment's contribution | most recommended | | Memory Soup | parameter-level weighted average | suits deep memory modules (DLA, Titans) | | Sparse Selective Caching (SSC) | MoE-style Top-k router over caches | most inference-efficient |

GRM's core formula:

\[y_t = γ_t^(s) * M_t^(s)(q_t) + Σ γ_t^(i) * M_L^(i)(q_t)\]

where γ depends on both the current token and historical segment content—selective reading of relevant history, not blind mixing.

A striking theoretical result (Section 4.1): with segment size 1 and value-less vector memory, Memory Caching is mathematically equivalent to gated global attention. The Transformer is an extreme special case of Memory Caching—a continuum:

N=1 → O(L²), equivalent to Transformer
N=L → O(L), pure RNN
N=L/C (constant segments) → O(L²/C), practical compromise
N=log L → O(L log L)

Experiments (760M parameters, 30B tokens):

Language modeling: Titans + GRM achieves Wiki ppl 19.14 vs Transformer++'s 24.18, beating it by +0.99% average accuracy.
Needle-in-a-Haystack (16K, hardest S-NIAH-3 UUID retrieval): DLA baseline 4.0% → DLA+GRM 18.2% (4.5×) → Titans+GRM 32.2% (near Transformer's 40.8%).
Contextual retrieval (SWDE, SQuAD, FDA): Titans+GRM averages 40.50 vs Transformer's 41.00 (baseline Titans: 31.75).

Sakana AI's CTM: bringing time back

Llion Jones—co-author of "Attention Is All You Need"—drastically reduced Transformer research, criticizing LLMs' "jagged intelligence": genius on some tasks, elementary mistakes on others. His diagnosis: we ignored time.

The Continuous Thought Machine (CTM) makes time an intrinsic dimension via an internal timeline decoupled from input, iterating from tick 1 to tick T:

1. Pretrained encoder: converts raw input to features (external component). 2. Synapse: a U-Net bridging input features and prior-tick activations into pre-activations. 3. Neuron-level models: every neuron has its own small MLP processing a history window (M=10~100 ticks)—each neuron has independent memory and temporal dynamics, not a simple ReLU. 4. Neural synchronization: a sync matrix S_t = Z_t × Z_t^T over all historical post-activations; randomly sampled entries form the latent representation used for attention input and output projection.

Adaptive compute: the model decides how long to think based on output confidence (entropy). Training computes loss at the lowest-loss tick and the highest-confidence tick, yielding graceful behavior—fast answers for easy inputs, more thought for hard ones.

Maze results: trained on 39×39 mazes (paths up to 100 steps), CTM generalizes to 99×99 mazes and longer paths while an LSTM baseline fails entirely. It exhibits "episodic future thinking"—planning by imagining future states, even without positional encodings. Ablations confirm neuron-level models and synchronization are jointly essential: removing either drops accuracy from 66% to below 50%.

Comparing the two paths

| Dimension | Memory Caching | CTM | |---|---|---| | Core motivation | Transformer efficiency | AI's missing time dimension | | Math basis | RNN + segment caching + gated aggregation | neural dynamics + synchronization + internal time | | Complexity | O(NL), interpolable to O(L²) or O(L) | RNN-like, variable internal ticks | | Relation to Transformer | unifying framework (Transformer is a special case) | outright replacement | | Inspiration | computer science (caching, routing) | neuroscience (oscillations, synchrony) | | Engineering maturity | high (plugs into existing RNNs) | medium (new training pipeline) | | Scale validation | 760M params, 30B tokens | small scale (9M-param maze) |

Efficiency: Memory Caching is the pragmatic choice—it enhances existing RNNs (Mamba, Titans, DLA), is validated at scale, and is compatible with current training infrastructure.
Cognition: CTM is more radical—the brain has no attention matrix; it communicates via synchronized neural oscillations. Attention is "brute-force search" over token pairs; CTM imitates dynamic, temporal retrieval instead.

Is the Scaling Law hitting a ceiling?

The Scaling Law doesn't answer whether architectural inefficiency offsets compute gains, or whether longer contexts help despite Context Rot. Memory Caching recalibrates compute-efficiency scaling; CTM redefines reasoning efficiency via adaptive computation. Future world models need continuous perception, long-term memory, dynamic planning, and causal reasoning—attention covers the first two, but has no "state" or temporal dynamics. Following Rich Sutton's "generative cognition" thesis—that intelligence generates predictive models rather than storing information—CTM's future thinking is an early step in that direction, while Memory Caching remains efficient retrieval.

Outlook

A plausible fusion: CTM's neuron-level models processing cached-segment dynamics, with GRM-style gating replacing CTM's random sampling. Timeline: Memory Caching could see production deployment within 2026; CTM likely needs 3-5 years for large-scale language model validation.

Conclusion: pick your architectural stance

The Transformer won't vanish tomorrow, but these papers mark the formal start of a post-Transformer era:

Choose Memory Caching if you care about efficiency, cost, deployability, and long contexts without changing training infrastructure.
Choose CTM if you care about cognition, temporal reasoning, and dynamic planning, and can accept engineering risk for a long-term vision.

Most likely: coexistence. Memory Caching becomes the mainstream cost-reduction alternative over 2-3 years, while CTM matures in robotics, real-time decisions, and games. The Transformer's reign won't end abruptly—but "Attention Is Not All You Need" is getting louder.

References:

1. Memory Caching: RNNs with Growing Memory (2026.02) - Ali Behrouz et al., Google Research. arXiv:2602.24281 2. Continuous Thought Machines (2025) - Luke Darlow, Llion Jones et al., Sakana AI. NeurIPS 2025 3. JetBrains Research (2025.12) - Smarter Context Management for LLM-Powered Agents, NeurIPS 2025 4. Chroma (2025) - Context Rot in Long-Context Retrieval 5. Rich Sutton (2025) - The Generative Turn in Cognitive Architecture