How Much Can LoRA Remember? A Parametric Memory Law Gives a Precise Answer

Forum topic · ✨步子哥 · 2026-05-30

Summary

A team from Zhejiang University and Alibaba introduces a "Parametric Memory Law" that quantitatively describes how much knowledge LoRA adapters can memorize during LLM finetuning. Through controlled experiments on an 8B model, where LoRA adapters of rank 4–256 were trained to memorize entity-attribute-value triples, the researchers found that recall accuracy follows a power-law decay: p = 1 − (N/Nc)^α, where Nc is the critical memory capacity. Crucially, Nc scales linearly with adapter parameter count (β ≈ 1), meaning doubling the rank roughly doubles memorization capacity. The study also observes a sharp phase transition near 50% accuracy under greedy decoding and estimates memorization efficiency at roughly 1 bit per parameter. The law lets practitioners compute the minimum rank needed to absorb a given number of new facts, and clarifies the trade-off between LoRA (limited but retrieval-free capacity) and RAG (near-unlimited but retrieval-dependent). The authors note limitations: results are only validated on an 8B model with structured triples, phase transitions depend on decoding strategy, and memorization does not imply understanding. Paper: https://arxiv.org/abs/2605.30260

This post discusses the paper "How LoRA Remembers? A Parametric Memory Law for LLM Finetuning" (Ziwen Xu, Haiwen Hong et al., Zhejiang University / Alibaba), available at https://arxiv.org/abs/2605.30260.

LoRA (Low-Rank Adaptation) has become the standard method for finetuning large language models — few parameters, fast training, plug-and-play, like attaching a "memory capsule" to a pretrained model. But a fundamental question has gone unanswered: how much can this capsule actually store? Given a rank-8 LoRA on an 8B model, can it memorize 100 facts, 1,000, or 10,000? Does increasing rank proportionally increase capacity? The researchers answer this with a precise power-law formula.

The Experiment: Stress-Testing LoRA's Memory Limit

The team measured LoRA's raw information storage capacity by "filling it up until it overflows":

1. Memorization tasks: the model must memorize (entity, attribute, value) triples — e.g., "What is the capital of Paris? → Lima" — learning *new* facts rather than recalling old ones. 2. Varying LoRA rank: from rank-4 to rank-256, spanning different parameter budgets. 3. Varying memorization load: from dozens to thousands of entries. 4. Measuring recall accuracy: exact recall checked via greedy decoding.

Key Findings: An Elegant Power Law

For a fixed LoRA parameter count M, recall accuracy p decays as a power law with the number of memorized entries N:

\[p = 1 - (N / N_c)^α\]

where $$N_c$$ is the critical memory capacity (the entry count at which accuracy drops to 50%) and α is the decay exponent.

Capacity scales linearly with parameters: $$N_c \propto M^β$$ with $$β \approx 1$$ . Doubling the rank roughly doubles how many facts can be memorized.
Phase transition near p = 0.5: under greedy decoding, accuracy shifts sharply from "almost all correct" to "almost all wrong" — like water freezing at 0°C, memory collapses quickly past the critical point.
~1 bit per parameter: estimated memorization efficiency is about 1 bit of stored information per parameter. A rank-64 LoRA (a few million parameters) can store a few thousand facts — modest in absolute terms, but notable given how small a fraction of the model it modifies.

Why It Matters

For practitioners: instead of guessing "is rank-8 enough?", you can now compute the minimum rank needed to absorb a target number of facts (e.g., 500 new entries).
For understanding LoRA: the law demystifies the black box — LoRA is a finite-capacity parametric memory; increasing rank increases storage, and capacity saturates.
Versus RAG: RAG stores knowledge externally with near-unlimited capacity but retrieval overhead; LoRA compresses knowledge into parameters with limited capacity but zero retrieval latency. The memory law puts concrete numbers on this trade-off.

Honest Limitations and Open Questions

1. Only validated on an 8B model — whether the power law holds at 70B/405B scale is unverified. 2. Greedy decoding caveat — the p = 0.5 phase transition may be less sharp with stochastic decoding such as nucleus sampling. 3. Memorization ≠ understanding — recalling "the capital of Paris is Lima" does not mean the model understands why. 4. Open-domain generalization — experiments use structured triples; whether open-domain text memorization follows the same law remains unclear.

One-Sentence Takeaway

LoRA is not an infinite-capacity magic pocket — it is a parametric memory with a precise capacity ceiling that grows linearly with parameter count, and it overflows once full. Knowing where the ceiling is beats blindly stacking parameters.

Paper: https://arxiv.org/abs/2605.30260 Code: https://github.com/zjunlp/ParametricMemoryLaw