HOLA: Giving Linear Attention a "Hippocampus" for Exact Memory
*An annotated walkthrough of "A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets" by Wanyun Cui (Shanghai University of Finance and Economics).*
The Brain's Lesson: Complementary Learning Systems
Imagine meeting "Professor Wang" at a dinner party. A month later you recognize his face but cannot recall his name or field—until you spot his vintage watch, and everything floods back. This illustrates that human memory is a layered system, not a video recorder:
- Compressed, gist-like memory ("he studies quantum computing")
- Exact, episodic memory (the watch's details)
- HOLA beats GDN by 16.1% on Wikitext and surpasses full-attention Transformer++.
- Retrieval: FDA 11.7 → 20.1 (+72%), SWDE 29.0 → 35.9 (+24%).
- RULER needle-in-a-haystack: GDN collapses to 0.14 at 32k; a recency-window variant only reaches 0.24; HOLA holds 0.58—generalizing to 16× the 2k training length.
- The ~16% perplexity advantage persists across 46M, 170M, and 340M scales.
- Surprise beats recency: at 32k, \(\beta\|e\|\) selection (0.58 recall) far outperforms a sliding window (0.24), because key facts are often distant.
- Ablations: RMSNorm-γ sharpening is the largest single lever (~2× multi-key capacity); the \(\beta\|e\|\) eviction rule beats recency at all lengths; larger caches show diminishing returns.
- Cost: only 12,480 trainable scalars (<0.004% of the model), ~31 MB inference memory (bf16), peak GPU allocation 0.75 GB vs GDN's 0.72 GB—flat regardless of context length.
- Cui, W. (2026). *A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets.* arXiv preprint arXiv:2607.02303.
- McClelland, McNaughton & O'Reilly (1995). Why there are complementary learning systems in the hippocampus and neocortex. *Psychological Review*, 102(3), 419.
- Katharopoulos et al. (2020). Transformers are RNNs. *ICML*.
- Gu & Dao (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752.
- Yang, Kautz & Hatamizadeh (2024). Gated delta networks. arXiv:2412.06464.
- Tse et al. (2007). Schemas and memory consolidation. *Science*, 316(5821), 76-82.
- Frankland & Bontempi (2005). The organization of recent and remote memories. *Nature Reviews Neuroscience*, 6(2), 119-130.
- Mohtashami & Jaggi (2023). Random-access infinite context length for transformers. arXiv:2305.16338.
- Hsieh et al. (2024). RULER: What's the real context size of your long-context language models? arXiv:2404.06654.
Neuroscience calls this Complementary Learning Systems (CLS): the neocortex learns slowly, compressing and generalizing, while the hippocampus records quickly and precisely for one-shot recall. The two must be separated—forcing one system to do both causes it to break down.
The paper's author asks: if the brain needs a hippocampus to complement the neocortex, does linear attention also lack a "hippocampus"?
Why Linear Attention "Forgets"
Softmax attention lets the model look back at every previous token—accurate but slow. Linear attention (and state-space models) instead compress history into a fixed-size state vector, like a single-page cheat sheet: fast, but lossy.
Linear attention updates its state with the delta rule:
where \(e_t = v_t - k_t^\top S_{t-1}\) is the residual (the part the state predicted wrong), and \(\beta_t\) is the write strength. The state's rank is capped at the key dimension \(d_k\). Once distinct key-value associations exceed that, new writes overwrite old ones. This is fatal for long-range facts—e.g., a birthday mentioned in paragraph 10 that is queried at paragraph 50. This is the needle-in-a-haystack problem, linear attention's Achilles' heel.
HOLA: A Hippocampus for the Recurrent State
HOLA augments the state with a small non-parametric cache:
| Component | Analogy | Function | |:---|:---|:---| | \(q_t^\top S_t\) | Neocortex | Parametric state, compressed gist | | \(g_t(q_t)\) | Hippocampus | Non-parametric cache, exact storage |
What to store: surprise
Rather than caching recent tokens, HOLA lets the delta rule itself decide. The score \(\beta \cdot \|e\|\) measures how much a token changed the state—i.e., how weakly the state "remembers" it. HOLA keeps the top-\(w\) tokens by this score (default \(w=64\)), regardless of position. Distant, surprising facts survive; forgettable ones don't.
How to read: from soft average to sharp retrieval
If the cache reuses unit-L2-norm queries/keys, logits are capped around 0.83, so even perfect matches get only ~3.5% of attention mass at \(w=64\)—another lossy soft average. HOLA applies a decoupled RMSNorm-γ sharpening on the cache path only:
Cache reads become near-argmax lookups, while state updates keep unit-norm stability.
Results (340M params, 15B SlimPajama tokens)
| Model | Wikitext PPL↓ | LAMBADA PPL↓ | |:---|---:|---:| | Transformer++ (full attention) | 26.88 | 42.15 | | GDN (linear baseline) | 27.32 | 30.95 | | HOLA | 22.92 | 30.26 |
Why It Works and What It Costs
Limitations and Future Work
1. The cache is bounded (~321 tokens), so very long or needle-dense contexts cannot retain everything (32k recall is 0.58, not 1.0). 2. A gap to full attention remains on pure token-extraction tasks. 3. The 340M results come from a single seed; no matched-memory comparison against learned eviction modules (e.g., LTE's CNN).
Future directions include scaling beyond 340M, learned eviction policies, multi-needle evaluation, and combinations with other memory architectures.
Takeaway
HOLA maps CLS theory onto architecture design at the functional level: state = neocortex, cache = hippocampus, \(\beta\|e\|\) = surprise-based memory consolidation, RMSNorm-γ = one-shot precise recall. It signals a shift from pure engineering optimization toward biologically-inspired design—and shows that sometimes the brain's billion-year-old solution translates remarkably well into AI.