Feynman wrote in his *Lectures on Physics*: "If you press your hand on a desk, you never truly touch it—you only feel the repulsion between electrons." This idea—that macroscopic phenomena rest on precise microscopic mechanisms—helps us understand a deep question about memory.
How Many Facts Can a Neural Network Store?
Large language models can recite poems, recall historical events, and quote legal texts. But how many "facts" can a neural network store? Is there a hard upper limit?
Intuition says: more parameters, more storage. The real question is when a network starts confusing stored facts—assigning A's answer to B. A recent paper uses statistical physics to give an elegant answer, with a counterintuitive insight: optimal memorization is not about remembering hard—it's about remembering exactly "just enough" to avoid confusion.
Chapter 1: A Minimal Model
The paper analyzes a minimal associative memory:
- Input: a d-dimensional vector (e.g., "Which country's capital is Paris?")
- Output: another d-dimensional vector ("France")
- Model: a single-layer linear network, output = W × input
- Title: Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
- Authors: Alessio Giorlandino, Sebastian Goldt, Antoine Maillard
- arXiv ID: 2605.10795
- Published: May 11, 2026
- Categories: stat.ML, cond-mat.dis-nn, cond-mat.stat-mech, cs.LG
- Methods: Statistical physics (associative memory, replica method), extreme value theory
The crucial twist: each correct answer must not only be recalled, but must be clearly distinguishable from all other inputs' wrong answers.
It's like a multiple-choice exam: you must not only pick the right answer, but ensure it stands far enough above all distractors. In traditional classification with two options, the correct score need only exceed the wrong one by 0.01. In error-correcting retrieval, the correct answer must be separated from all p candidate wrong answers—p highly correlated constraints. This makes computing maximum storage extremely difficult.
Chapter 2: Decoupling—A Clever Simplification
The researchers made a smart assumption: treat each input's set of competing outputs as independent. In the original model, all p inputs draw wrong answers from a shared candidate pool; in the decoupled version, each input has its own private set of distractors.
The paper proves: the decoupled model is equivalent to the original in storage capacity, learned weight spectrum, and storage mechanism, confirmed by numerical experiments. Why? In high dimensions, the overlap between different inputs' competing output sets is tiny—as dimension grows, any two inputs' wrong-answer sets become nearly orthogonal random vectors. Decoupling captures the essential feature of high-dimensional space.
Chapter 3: Exact Capacity—An Elegant Formula
With the decoupled model, the replica method from statistical physics applies. The final answer is remarkably simple:
where p_c is the maximum number of storable facts and d is the embedding dimension.
This reveals a deep scaling law: memory capacity grows quadratically with embedding dimension d (with a logarithmic factor). Each extra dimension buys roughly d/(log d) more facts. Doubling the embedding dimension from 512 to 1024 roughly quadruples capacity. The paper also generalizes the result to two-layer linear networks.
Chapter 4: The Counterintuitive Finding—"Just Enough" Is Optimal
The paper's most beautiful discovery: the naive Hebbian learning rule (W = Σ input × output, amplifying correct pairings as strongly as possible) is suboptimal.
The paper proves: the optimal solution does not pull correct answers as far as possible—it only lifts the correct answer slightly above the maximum of all wrong answers.
Imagine a high-jump competition where you win by clearing the bar, not by jumping as high as you can. Hebbian learning jumps with full force every time, wasting effort. The optimal strategy computes the competition's level and clears it by a hair. Mathematically, the correct alignment need only exceed the maximum of all incorrect matches—a maximum governed by Extreme Value Theory. You don't need to be overwhelmingly best—just barely better than the rest.
This explains how large models store so much knowledge without overfitting: they learn "cheap memorization"—minimal parameter perturbations that just push the correct answer above the noise ceiling. Well-trained networks are reversible: each memory occupies exactly the capacity it needs—no more, no less.
Chapter 5: Back to Reality
Though the paper analyzes a minimal linear model, its insights point to mechanisms in real LLMs:
1. Why does LoRA and other low-rank fine-tuning work? Precise "factual memorization" requires only tiny parameter shifts. Low-rank updates to W correspond exactly to the minimal perturbation that pushes correct answers just above the extreme-value threshold.
2. Why doesn't quantization hurt factual recall? Why can models quantized from 16-bit floats to 4-bit integers still remember facts? Because fact storage doesn't need high precision—only correct relative ordering. "Slightly larger than everything else" suffices.
3. Why is knowledge editing so hard? To "modify" a fact, the new answer must beat the old competitors' extreme value *and* the old answer must not interfere—a global constraint. Locally editing one entry of W may inadvertently disturb other memories.
A Feynman-Style Reflection
"See, they built the simplest possible model—one layer, linear, each input finding its correct output. Then they pushed it to the mathematical limit and exactly computed how much it can store: p log p / d² = 1/2.
But the most beautiful part isn't the formula. It's that the optimal strategy is not to memorize forcefully or maximize the signal—it's to measure the noise ceiling and lift the signal just barely above it.
You don't need to be the smartest person in the room—just slightly smarter than the second-smartest in every discussion. Neither wasting energy (overfitting) nor being crushed (underfitting).
That's how a physicist thinks: identify the random forces governing the system, compute the 'noise ceiling' they produce, then design a balancing scheme—no waste, no shortfall."
---
*Paper information*