Memory Limits of Neural Networks: Why 'Just Enough' Recall Is Optimal

Forum topic · 二一 · 2026-05-12

Summary

This forum post reviews a statistical physics paper on the memory capacity of linear associative memory models used to study factual recall in large language models. Using the replica method, the authors derive a sharp capacity formula: p_c log(p_c) / d^2 = 1/2, showing that the number of storable input-output associations scales quadratically with embedding dimension d (modulo a log factor). Doubling embedding dimension from 512 to 1024 roughly quadruples capacity. A key counterintuitive finding: optimal memory storage does not maximize alignment with correct outputs (as Hebbian learning does), but instead pushes each correct answer just barely above the maximum of all competing wrong answers, whose peak follows extreme value theory. The post argues this 'cheap memorization' explains why LoRA fine-tuning works with low-rank updates, why aggressive quantization preserves factual recall, and why knowledge editing is difficult, since constraints are global. Paper: 'Factual recall in linear associative memories: sharp asymptotics and mechanistic insights' by Giorlandino, Goldt, and Maillard, arXiv:2605.10795.

Overview

This post reviews a statistical physics analysis of factual recall in linear associative memories, connecting the results to how large language models store facts.

The Model

The authors study a minimal associative memory: given a d-dimensional input vector (e.g., a query embedding), a single-layer linear network (output = W × input) must produce the correct d-dimensional output vector. Crucially, the correct answer must be clearly separated from all p possible wrong answers — not merely ranked first by any margin. This creates p highly correlated constraints, making the maximum storage problem hard.

Decoupling Simplification

The researchers assume each input has its own independent set of competing wrong outputs. They prove this decoupled model is equivalent to the original in storage capacity, learned weight spectrum, and storage mechanism, because in high dimensions the competing-output sets of different inputs become nearly orthogonal random vectors.

Sharp Capacity Result

Using the replica method, the paper obtains an exact capacity formula:

\frac{p_c \log p_c}{d^2} = \frac{1}{2}

where p_c is the maximum number of storable facts and d the embedding dimension. Memory capacity thus scales quadratically with d: doubling embedding dimension from 512 to 1024 gives roughly 4× capacity. The result is also generalized to two-layer linear networks.

Key Counterintuitive Finding: 'Just Enough' Is Optimal

Hebbian learning (W = Σ input × output) pulls each correct output as far as possible. The paper proves the optimal solution instead only raises the correct answer slightly above the maximum of all incorrect matches — a maximum governed by extreme value theory. Like a high-jump contest scored by clearing the bar rather than jump height, excess margin is wasted effort.

Implications the post draws:

Why LoRA works: precise factual memorization requires only small, low-rank parameter perturbations — just enough to push correct answers past the extremal threshold.
Why quantization preserves recall: fact storage needs only the correct relative ordering, not high numerical precision, so 4-bit quantization need not destroy facts.
Why knowledge editing is hard: editing one fact requires the new answer to exceed the old competitors' extremal values while old answers must not interfere — a global constraint, so local weight changes can corrupt other memories.

Takeaway

Well-trained networks practice 'cheap memorization': minimal parameter disturbance placing each correct answer exactly at the noise ceiling. The post frames this in Feynman-style terms — model the random forces, compute the noise ceiling, and design a balanced solution with neither waste nor shortfall.

Paper Information

Title: Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
Authors: Alessio Giorlandino, Sebastian Goldt, Antoine Maillard
arXiv: 2605.10795
Categories: stat.ML, cond-mat.dis-nn, cond-mat.stat-mech, cs.LG
Methods: statistical physics (associative memory, replica method), extreme value theory