Titans and MIRAS: Google's New Architectures Give AI Long-Term Memory up to 2 Million Tokens

Forum topic · ✨步子哥 · 2025-12-07

Summary

This zhichai.net forum post explains Google Research's Titans and MIRAS frameworks, presented around NeurIPS 2025, which tackle the quadratic O(N²) cost of Transformer self-attention in long-context settings. Titans introduces a Neural Long-Term Memory Module—an MLP updated dynamically at test time (Test-Time Training)—with a 'Memory as Context' (MAC) variant that compresses history into summaries fed to attention layers, and a 'Surprise Metric' (approximated via KL divergence) that decides what deserves long-term storage. MIRAS provides a unified framework decomposing sequence models into four design choices: memory architecture, attention bias, retention gate, and memory algorithm, yielding variants YAAD, MONETA, and MEMORA. Reported results show context scaling to 2 million tokens, outperforming baselines like Mamba 2. The post also touches on Jeff Dean's NeurIPS 2025 remarks about Google's open-sourcing of the Transformer.

Overview

This post from zhichai.net discusses two Google Research architectures—Titans and MIRAS—aimed at solving the long-context memory problem in Transformers. The central claim: by combining the speed of recurrent models with attention's global insight, and by learning *at test time*, AI can scale context windows to 2 million tokens.

Key points

1. The Transformer bottleneck

Self-attention computes $Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$ ; at N = 2 million tokens, the $$QK^T$$ matrix requires O(N²) space and time.
Existing alternatives trade performance for speed:
Linear RNNs compress context into a fixed-size state (O(N)) but lose details.
State-space models (SSMs) improve information flow but still miss long-range dependencies.

2. Titans: dynamic long-term memory

Core component: a Neural Long-Term Memory Module—a multi-layer perceptron (MLP) updated dynamically at inference time via Test-Time Training, rather than a static recurrent vector.
MAC (Memory as Context) variant: distills long-term memory into a historical summary passed to attention layers alongside short-term input, preserving key context without re-reading everything.
Surprise Metric: quantifies deviation of new input from stored memory, e.g. $Surprise = D_{KL}(P_{new} \| P_{memory})$ ; low surprise → short-term storage, high surprise → committed to long-term memory.
Reported to scale to 2 million tokens, beating baselines on needle-in-a-haystack tasks.

3. MIRAS: a unified framework for sequence modeling

Memory architecture

Attention bias

Retention gate

Gate = \sigma(W \cdot [h_t, m_{t-1}])

Memory algorithm

Derived models:

YAAD: uses Huber Loss for robustness to outliers.
MONETA: generalized norms to tightly regulate attention and forgetting.
MEMORA: probabilistic (Bayesian) memory updates, $P(m_t | x_t) \propto P(x_t | m_{t-1}) P(m_{t-1})$ .

The post reports these models outperform linear-recurrent rivals like Mamba 2 and rival much larger models on very long contexts.

4. Industry context

At NeurIPS 2025, Jeff Dean, asked by narrative whether Google regretted publishing the Transformer (a jab referencing a Hinton remark), answered no—sharing it benefited the world.
The post frames Titans + MIRAS as part of Google's broader momentum (Nano Banana, Gemini 3 Pro).

Takeaway

The post argues the future of AI memory lies in adaptive, test-time learning—models that selectively store surprising information and compress the rest—rather than ever-larger attention matrices. Suggested applications include legal document review, longitudinal medical records, and conversational agents with lifelong memory.

References

1. Titans & MIRAS: Helping AI Have Long-Term Memory — Google Research Blog 2. Titans: A New Architecture for Long-Term Memory in Sequence Models — arXiv:2501.00663 3. MIRAS: A Unified Framework for Sequence Modeling — arXiv:2504.13173 4. Transformer Limitations and Beyond: A Survey on Long-Context Modeling — NeurIPS 2025 Proceedings 5. Test-Time Training: Enabling Adaptive Inference in LLMs — Google DeepMind Whitepaper