Google's Titans & MIRAS: Breaking the AI Long-Term Memory Bottleneck
This forum post is an infographic-style overview of two Google Research works: *Titans: Learning to Memorize at Test Time* and *It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization* (MIRAS).
Current AI Memory Limitations
- Transformers slow down and become more expensive as text length grows, with quadratic computational complexity.
- Existing models (ChatGPT, Claude, Gemini) tend to forget information in long conversations or long documents.
- This is a long-standing foundational limitation that hinders progress toward AGI.
- Short-term memory (core): Handles the current task via attention mechanisms, analogous to human short-term memory.
- Long-term memory: Stores historical information, learns and updates at runtime, using a "surprise metric" to decide what to store.
- Persistent memory: Knowledge fixed during training, stable and unchanging, analogous to meta-memory.
- Lower perplexity (better performance) on long sequences; deep memory architectures maintain advantages over Mamba across parameter scales.
- Strong results on Babai Long (fact-based QA over extremely long documents).
- Even smaller, cheaper models retain high accuracy on long sequences.
- Significant advantages over GPT-4 and Mamba, especially at very long context lengths.
- Legal document analysis (very long legal files)
- Medical record management (long-term patient data)
- Scientific paper understanding (cross-paper concept connections)
- Codebase maintenance (large projects)
- Personal assistants (remembering user preferences and history)
- AGI significance: a human-brain-like memory system enabling previously impossible applications
- Titans and MIRAS address the root problem of AI long-term memory.
- They enable test-time learning, letting models evolve during use.
- They offer a new technical path toward AGI, with Google again leading architecture innovation beyond the Transformer it created.
Titans: A Brain-Inspired Three-Tier Memory System
The MAC (Memory as Context) architecture treats memory as context, enabling context windows exceeding 2 million tokens.
MIRAS: A Unified Framework for Sequence Models
MIRAS reveals that major AI architectures (Transformer, RNN, Mamba, etc.) are essentially the same at their core, opening the door to better memory system design. Its four components:
1. Memory architecture — how information is stored (vectors, matrices, MLPs). 2. Attentional bias — the internal learning objective that determines what the model prioritizes. 3. Retention gating — a forgetting mechanism balancing learning new knowledge vs. retaining old memories. 4. Memory algorithm — the optimization algorithm for updating the memory state.
Innovation: a non-Euclidean objective function allowing more complex mathematical penalty mechanisms.