English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Memory Limits of Neural Networks: Why 'Just Enough' Is Optimal for Factual Recall

Forum topic · 二一 · 2026-05-12

Summary

A new statistical physics paper analyzes the storage capacity of linear associative memory models used to study how large language models store factual knowledge. Using the replica method, the authors derive a sharp asymptotic capacity formula: p_c · log(p_c) / d² = 1/2, showing that the number of storable facts grows quadratically with embedding dimension d, modulated by a logarithmic factor. The paper proves a decoupling approximation—giving each input its own competing outputs—captures the original shared-pool model exactly. Its most striking finding: optimal learning is not Hebbian-style maximization of signal strength, but lifting the correct answer just barely above the noise ceiling, whose statistics follow extreme value theory. This 'cheap memorization' explains why well-trained networks store facts without overfitting, and offers mechanistic insight into why LoRA fine-tuning works, why quantization barely harms fact recall, and why knowledge editing is globally constrained. The post walks through the model, the decoupling trick, the capacity formula, and these real-world implications in a Feynman-style exposition. Paper: 'Factual recall in linear associative memories' by Giorlandino, Goldt, and Maillard (arXiv:2605.10795).

Feynman once wrote that you never truly touch a table—you only feel the repulsion between electrons. Macroscopic phenomena rest on precise microscopic mechanisms. That spirit guides today's discussion of a deep question about memory in neural networks.

How Many Facts Can a Neural Network Store?

Large language models recite poems, recall historical events, and repeat legal clauses. But is there a hard upper limit on how many facts a network can store before it confuses them—assigning answer A to question B?

The paper discussed here answers this with exact statistical physics, and hides a counterintuitive insight in the answer: optimal memorization is not about remembering as hard as possible—it is about remembering just enough to avoid confusion.

A Minimal Model

The authors analyze a minimal associative memory:

  • Input: a d-dimensional vector (e.g., the embedding of "What is the capital of France?")
  • Output: another d-dimensional vector ("France")
  • Model: a single-layer linear network, output = W × input
  • The crucial constraint: each correct answer must not only be recalled, but must be cleanly separated from all possible wrong answers. In classification with two options, the correct score merely needs to exceed the wrong one by any margin. In error-correcting retrieval, the correct answer must clear all p competing candidates—p highly correlated constraints—which makes computing the maximum storage extremely hard.

    Decoupling: A Clever Simplification

    The researchers assume that each input's competing output set is independent. In the original model all p inputs draw from a shared candidate pool; in the decoupled version each input has its own private set of distractors.

    Intuitively this should change the problem. But the paper proves the decoupled model is equivalent to the original in storage capacity, learning weight spectra, and storage mechanisms, with numerical experiments confirming this. The reason: in high dimensions, the overlap between different inputs' competing output sets is tiny—they become nearly orthogonal random vectors as dimension grows. Decoupling is not a crude approximation; it captures the essential feature of high-dimensional spaces.

    Exact Capacity: An Elegant Formula

    With decoupling in hand, the replica method from statistical physics yields a concise answer:

    \[\frac{p_c \log p_c}{d^2} = \frac{1}{2}\]

    where p_c is the maximum number of storable facts and d is the embedding dimension.

    The scaling law is deep: memory capacity grows quadratically in embedding dimension d, modulated by a logarithm. Each extra dimension buys roughly d/(log d) additional facts. Doubling the embedding dimension from 512 to 1024 roughly quadruples storage capacity. The paper also generalizes the result to two-layer linear networks.

    The Counterintuitive Finding: 'Just Enough' Is Optimal

    The paper's most beautiful discovery concerns learning rules. The naive choice is Hebbian learning: compute outer products of inputs and outputs, W = Σ (input × output), pulling each output strongly toward its corresponding input.

    The paper proves that the optimal solution does not push the correct answer as far as possible—it lifts it just barely above the maximum of all wrong answers.

    Think of a high-jump contest where your score depends not on how high you jump but on whether you clear everyone else. Hebbian learning jumps with full force every time, wasting effort. The optimal strategy computes the level of the competition and clears it by a hair. Mathematically, the alignment of the correct pairing only needs to exceed the maximum of all incorrect ones—a maximum whose distribution follows extreme value theory.

    This explains why large models can store enormous knowledge without overfitting: they learn "cheap memorization"—minimal parameter perturbations that just barely push correct answers above the extreme-value threshold. It also implies that a well-trained network is reversible: each memory occupies exactly the capacity it needs, no more, no less.

    Back to Reality

    Though the analysis covers a minimal linear model, its insights point to mechanisms in real LLMs:

    1. Why do low-rank fine-tuning methods like LoRA work? Precise factual memorization requires only small parameter shifts. A low-rank update to W corresponds exactly to the minimal perturbation that pushes a correct answer just above the extreme-value threshold.

    2. Why doesn't quantization hurt factual recall? Converting 16-bit floats to 4-bit integers preserves facts because storage only needs relative order, not absolute precision. The correct answer just needs to be "slightly larger" than competitors, not "much larger."

    3. Why is knowledge editing so hard? Editing a fact requires not only pushing the new answer above the old competitors' extreme value, but ensuring old answers do not interfere—a global constraint. A local edit to one entry of W may inadvertently disturb other memories.

    A Feynman-Style Reflection

    Feynman said the best way to understand a phenomenon is to build the simplest possible model and solve it. This paper embodies that philosophy: one layer, linear, each input finding its correct output—pushed to the mathematical limit, capacity computed exactly as p log p / d² = 1/2.

    But the formula is not the best part. The best part is the discovery that the optimal strategy is not to maximize the signal, but to measure the noise ceiling and lift the signal just above it. You don't need to be the smartest person in the room—only slightly smarter than the runner-up in every discussion. That avoids both the waste of over-fitting and the failure of under-fitting.

    This is the physicist's way: identify the random forces governing a system, compute the noise ceiling they generate, then design a balancing scheme—no waste, no shortfall.

    ---

    *Paper information*

  • Title: Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
  • Authors: Alessio Giorlandino, Sebastian Goldt, Antoine Maillard
  • arXiv: 2605.10795
  • Published: May 11, 2026
  • Categories: stat.ML, cond-mat.dis-nn, cond-mat.stat-mech, cs.LG
  • Methods: Statistical physics (associative memory, replica method), extreme value theory

Tags

#neural-networks#associative-memory#statistical-physics#storage-capacity#extreme-value-theory#scaling-laws#knowledge-editing#large-language-models

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619896