Your AI Is Leonard from Memento: Why Continual Learning Is the Next Frontier for LLMs

After reading a16z's *Why We Need Continual Learning*, one image came to mind: Leonard Shelby from Nolan's *Memento*—a man who reboots every few minutes, surviving only on Polaroids, tattoos, and notes, living by the creed "trust facts, not memory." That is exactly today's large language models: trained on the internet, parameters frozen, living in an eternal present, pretending to remember via chat logs, retrieval systems, and system prompts. a16z's Malika Aubakirova and Matt Bornstein call these amnesic models. The core claim: we wrongly assume In-Context Learning has solved learning—it hasn't.

Key points

1. The Filing Cabinet Fallacy

• Ilya Sutskever: AGI and pretraining miss the target—humans have base skills but lack knowledge, and we rely on continual learning. Deployment itself includes learning; it's a process, not a delivered product.
• Giving a model unlimited storage and retrieval is not learning. Learning's essence is lossy compression into parameters, forcing structure discovery and transferable representations. Post-deployment, we swapped compression for external memory.
• Rich Sutton's Bitter Lesson echoes: learning from compute and data ultimately beats hand-designed external scaffolding.

2. Some things can't be written into a context window

• Citing Yu Sun: Fermat's Last Theorem went unsolved for 350 years not for lack of literature but because the solution required a conceptual leap—Wiles bridged elliptic curves and modular forms in near-isolation. Such breakthroughs are invented, not retrieved.
• In-context learning already fails at: genuinely novel discovery, adversarial settings, and tacit knowledge too subtle for language (radiology texture intuition, speaker-specific audio micro-dynamics).
• ChatGPT's memory feature feels creepy because users want competence, not recollection: "I remember what you said" (retrieval) vs. "I understand how you think" (learning).

3. A spectrum: Context → Modules → Weights

• Context: mature but depth-capped by context length; multi-agent architectures approximate longer memory via agents passing notes.
• Modules: partial compression (compressed KV caches, adapters, external memory). An 8B model with the right modules can match a 109B model on specific tasks with little extra memory.
• Weights: true parametric learning, via five directions:

| Direction | Representative work | One-liner | |---|---|---| | Regularization / weight-space | EWC, weight interpolation | Lock important parameters or blend old/new weight configurations | | Test-time training (TTT) | TTT layers, TTT-E2E, TTT-Discover | Run gradient descent at inference, compressing new info on the spot | | Meta-learning | MAML, Nested Learning | Train models that learn how to learn | | Distillation | LoRD, SDFT | Self-distillation to avoid forgetting | | Recursive self-improvement | STaR, AlphaEvolve | Bootstrap reasoning from self-generated reasoning |

• Google's Nested Learning argues that splitting forward structure from optimizer ("brain" vs "teacher") is an illusion: the model is a set of nested micro-optimizations, with Fast Weights adapting at inference and Slow Weights consolidating long-term memory.

4. Why you can't just fine-tune

• Catastrophic forgetting (stability–plasticity dilemma)
• Time-decoupling: invariant rules and mutable state are entangled in the same weights
• Failed logical integration: updating one fact doesn't propagate to its logical consequences
• No way to forget: there is no differentiable subtraction for wrong or harmful knowledge
• Safety: alignment may degrade unpredictably after narrow fine-tuning; continual updates create a slow, persistent data-poisoning surface rooted in weights; models become moving targets for versioning, regression testing, and certification; privacy risks grow as interactions are compressed into parameters. These are open problems, not fundamental impossibilities.

5. 2026: the year of continual learning

• TTT-E2E (Astera Institute et al., arXiv 2512.23675): self-supervised learning over read tokens before prediction—encoding context into weights, not KV cache. 2.7× faster than full-attention Transformers on 128K contexts.
• Nested Learning + HOPE (Google): a continuum memory system with self-referential dynamic correction.
• SDFT (MIT / ETH Zurich): self-distillation fine-tuning where the model is both student and teacher; 89% strict new-knowledge accuracy, 98% out-of-distribution generalization.
• The Era of Experience (Silver & Sutton): moving from human-data imitation to learning from the agent's own actions and feedback; AlphaProof generated hundreds of millions of proofs after learning from 100K human ones.

Conclusion

Leonard's tragedy was not incompetence but the inability to compound experience—everything stayed external. Retrieval is not learning. The path forward is layering: context for first-line adaptation, modules for personalization, weight-level learning for discovery, adversarial adaptation, and tacit knowledge. Perhaps we'll need to redefine "a model" as an evolving system of memory, update algorithms, and abstraction. 2026 is when the car starts moving.

Sources

• Why We Need Continual Learning — Malika Aubakirova, Matt Bornstein, a16z (2026-04-22)
• EWC — Kirkpatrick et al., PNAS 2017
• MAML — Finn et al., ICML 2017
• TTT-E2E — End-to-End Test-Time Training for Long Context, arXiv 2512.23675 (2025-12)
• Nested Learning — Google Research, 2025
• SDFT — Shenfeld et al., MIT/ETH Zurich, 2026
• Era of Experience — David Silver & Richard S. Sutton, DeepMind, 2025-04
• Survey — Continual Learning in Large Language Models, arXiv 2603.12658 (2026-03)