Google's Titans and MIRAS: Giving AI Long-Term Memory Beyond the Transformer Bottleneck

*English adaptation of a zhichai.net forum post discussing Google Research's Titans and MIRAS architectures presented at NeurIPS 2025.*

The Transformer bottleneck

Since its 2017 introduction at Google, the Transformer has dominated AI, but its self-attention mechanism scales quadratically: memory and compute cost O(N²) as sequence length grows. The core computation is:

At N = 2 million tokens, the QK^T matrix alone becomes prohibitive, especially for resource-constrained devices. Prior workarounds include:

• Linear recurrent networks (RNNs): compress context into a fixed-size state for O(N) scaling, but lose detail.
• State space models (SSMs): improve information flow but still miss fine-grained dependencies.

Titans and MIRAS aim to combine RNN speed with Transformer-level insight via Test-Time Training—learning during inference rather than relying on static weights.

Titans: a neural long-term memory

Titans centers on a Neural Long-Term Memory Module: instead of a static state vector, it is a multi-layer perceptron (MLP) whose weights update dynamically at test time, requiring no offline retraining.

Key ideas:

• MAC (Memory as Context) variant: long-term memory is distilled into a historical summary fed to the attention layer alongside current short-term input, so the model retrieves relevant fragments instead of re-reading everything.
• Surprise metric: inspired by human psychology, the model quantifies deviation of new input from existing memory, e.g. \(Surprise = D_{KL}(P_{new} \| P_{memory})\). Low-surprise tokens stay in short-term storage; high-surprise tokens are written into long-term memory.

Experiments reported in the post claim Titans scales to 2 million tokens and beats baseline models on needle-in-a-haystack retrieval tasks, with use cases such as legal document review and lifetime medical records analysis.

MIRAS: a unified framework for sequence modeling

MIRAS reframes sequence models (RNNs, Transformers, and beyond) around four design pillars:

1. Memory architecture — vector state vs. an MLP. 2. Attention bias — the internal objective determining what the model prioritizes. 3. Retention gate — how forgetting is balanced against retention, e.g. \(Gate = \sigma(W \cdot [h_t, m_{t-1}])\). 4. Memory algorithm — the update rule, from MSE to non-Euclidean objectives robust to noise.

Three derived models are highlighted:

• YAAD: uses Huber loss to handle outliers gracefully.
• MONETA: uses generalized norms to tightly regulate attention and forgetting.
• MEMORA: Bayesian-style memory updates, \(P(m_t | x_t) \propto P(x_t | m_{t-1}) P(m_{t-1})\).

The post reports these models outperform linear-recurrent rivals like Mamba 2 and, at small parameter counts, approach GPT-4-level performance—particularly on very long contexts.

Closing notes

The post ends with an anecdote from NeurIPS 2025: asked by Geoff Hinton whether Google regrets publishing the Transformer, Jeff Dean answered no—it has benefited the world. The author frames Titans + MIRAS as evidence that open sharing accelerates AI, and speculates on future directions such as hybrid quantum or brain–computer-interface memory systems.

Key points

• Transformer self-attention is O(N²), making million-token contexts impractical.
• Titans uses a test-time-trained MLP as long-term memory; the MAC variant feeds memory summaries into attention.
• A KL-divergence-based surprise metric governs what is remembered long-term.
• MIRAS unifies sequence models via four design choices (memory architecture, attention bias, retention gate, memory algorithm).
• Derived models YAAD, MONETA, and MEMORA reportedly beat Mamba 2 and rival much larger models on long-context tasks.

References

1. Titans & MIRAS: Helping AI Have Long-Term Memory — Google Research Blog 2. Titans: A New Architecture for Long-Term Memory in Sequence Models — arXiv:2501.00663 3. MIRAS: A Unified Framework for Sequence Modeling — arXiv:2504.13173 4. Transformer Limitations and Beyond: A Survey on Long-Context Modeling — NeurIPS 2025 Proceedings 5. Test-Time Training: Enabling Adaptive Inference in LLMs — Google DeepMind Whitepaper