Feynman once wrote that you never truly touch a table—you only feel the repulsion between electrons. Macroscopic phenomena rest on precise microscopic mechanisms. That spirit guides today's discussion of a deep question about memory in neural networks.
How Many Facts Can a Neural Network Store?
Large language models recite poems, recall historical events, and repeat legal clauses. But is there a hard upper limit on how many facts a network can store before it confuses them—assigning answer A to question B?
The paper discussed here answers this with exact statistical physics, and hides a counterintuitive insight in the answer: optimal memorization is not about remembering as hard as possible—it is about remembering just enough to avoid confusion.
A Minimal Model
The authors analyze a minimal associative memory:
- Input: a d-dimensional vector (e.g., the embedding of "What is the capital of France?")
- Output: another d-dimensional vector ("France")
- Model: a single-layer linear network, output = W × input
- Title: Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
- Authors: Alessio Giorlandino, Sebastian Goldt, Antoine Maillard
- arXiv: 2605.10795
- Published: May 11, 2026
- Categories: stat.ML, cond-mat.dis-nn, cond-mat.stat-mech, cs.LG
- Methods: Statistical physics (associative memory, replica method), extreme value theory
The crucial constraint: each correct answer must not only be recalled, but must be cleanly separated from all possible wrong answers. In classification with two options, the correct score merely needs to exceed the wrong one by any margin. In error-correcting retrieval, the correct answer must clear all p competing candidates—p highly correlated constraints—which makes computing the maximum storage extremely hard.
Decoupling: A Clever Simplification
The researchers assume that each input's competing output set is independent. In the original model all p inputs draw from a shared candidate pool; in the decoupled version each input has its own private set of distractors.
Intuitively this should change the problem. But the paper proves the decoupled model is equivalent to the original in storage capacity, learning weight spectra, and storage mechanisms, with numerical experiments confirming this. The reason: in high dimensions, the overlap between different inputs' competing output sets is tiny—they become nearly orthogonal random vectors as dimension grows. Decoupling is not a crude approximation; it captures the essential feature of high-dimensional spaces.
Exact Capacity: An Elegant Formula
With decoupling in hand, the replica method from statistical physics yields a concise answer:
where p_c is the maximum number of storable facts and d is the embedding dimension.
The scaling law is deep: memory capacity grows quadratically in embedding dimension d, modulated by a logarithm. Each extra dimension buys roughly d/(log d) additional facts. Doubling the embedding dimension from 512 to 1024 roughly quadruples storage capacity. The paper also generalizes the result to two-layer linear networks.
The Counterintuitive Finding: 'Just Enough' Is Optimal
The paper's most beautiful discovery concerns learning rules. The naive choice is Hebbian learning: compute outer products of inputs and outputs, W = Σ (input × output), pulling each output strongly toward its corresponding input.
The paper proves that the optimal solution does not push the correct answer as far as possible—it lifts it just barely above the maximum of all wrong answers.
Think of a high-jump contest where your score depends not on how high you jump but on whether you clear everyone else. Hebbian learning jumps with full force every time, wasting effort. The optimal strategy computes the level of the competition and clears it by a hair. Mathematically, the alignment of the correct pairing only needs to exceed the maximum of all incorrect ones—a maximum whose distribution follows extreme value theory.
This explains why large models can store enormous knowledge without overfitting: they learn "cheap memorization"—minimal parameter perturbations that just barely push correct answers above the extreme-value threshold. It also implies that a well-trained network is reversible: each memory occupies exactly the capacity it needs, no more, no less.
Back to Reality
Though the analysis covers a minimal linear model, its insights point to mechanisms in real LLMs:
1. Why do low-rank fine-tuning methods like LoRA work? Precise factual memorization requires only small parameter shifts. A low-rank update to W corresponds exactly to the minimal perturbation that pushes a correct answer just above the extreme-value threshold.
2. Why doesn't quantization hurt factual recall? Converting 16-bit floats to 4-bit integers preserves facts because storage only needs relative order, not absolute precision. The correct answer just needs to be "slightly larger" than competitors, not "much larger."
3. Why is knowledge editing so hard? Editing a fact requires not only pushing the new answer above the old competitors' extreme value, but ensuring old answers do not interfere—a global constraint. A local edit to one entry of W may inadvertently disturb other memories.
A Feynman-Style Reflection
Feynman said the best way to understand a phenomenon is to build the simplest possible model and solve it. This paper embodies that philosophy: one layer, linear, each input finding its correct output—pushed to the mathematical limit, capacity computed exactly as p log p / d² = 1/2.
But the formula is not the best part. The best part is the discovery that the optimal strategy is not to maximize the signal, but to measure the noise ceiling and lift the signal just above it. You don't need to be the smartest person in the room—only slightly smarter than the runner-up in every discussion. That avoids both the waste of over-fitting and the failure of under-fitting.
This is the physicist's way: identify the random forces governing a system, compute the noise ceiling they generate, then design a balancing scheme—no waste, no shortfall.
---
*Paper information*