> Source: *How LoRA Remembers? A Parametric Memory Law for LLM Finetuning*, Ziwen Xu, Haiwen Hong et al., Zhejiang University / Alibaba, https://arxiv.org/abs/2605.30260
---
1. Introduction: LoRA as a "Memory Capsule"
LoRA (Low-Rank Adaptation) has become the standard for LLM finetuning: fewer parameters, faster training, plug-and-play—like attaching a "memory capsule" to a large model to store new knowledge.
But a question has never been clearly answered: how much can this capsule hold?
If you finetune an 8B model with a rank-8 LoRA, how many facts can it memorize—100? 1,000? 10,000? How much more capacity does each rank increase buy? Is there an exact formula?
The Zhejiang University / Alibaba team says yes. They discovered the Parametric Memory Law—a power-law formula describing the relationship between LoRA's memory capacity and its parameter count.
---
2. Experimental Design: Brute-Force Testing LoRA's Memory Limits
The most direct way to measure LoRA's memory is to keep stuffing facts in until it overflows.
The team's setup:
1. Construct memorization tasks: the model must remember a set of (entity, attribute, value) triples, e.g., "What is the capital of Paris? → The capital of Paris is Lima" (storing *new* facts, not recalling old knowledge). 2. Vary LoRA rank: from rank-4 to rank-256, covering different parameter scales. 3. Vary the number of memory entries: from dozens to thousands, increasing gradually. 4. Measure recall accuracy: use greedy decoding to check whether the model can recall each entry exactly.
The key to this design is its "brute-force" nature—rather than benchmarking performance on a particular task, it directly measures LoRA's information storage capacity.
---
3. Core Finding: An Elegant Power Law
The experiments revealed a clear pattern:
When LoRA's parameter count M is fixed, recall accuracy p decays as a power law with the number of memorized entries N:
where \(N_c\) is the "critical memory capacity"—the number of entries at which accuracy drops to 50%—and α is the decay exponent.
1. Critical capacity scales linearly with parameter count
\(N_c ∝ M^β\) with \(β ≈ 1\). This means LoRA's memory capacity grows roughly linearly with its parameter count: doubling the rank roughly doubles the number of memorizable entries.
2. Phase transition at p = 0.5
Under greedy decoding, the accuracy curve shows a phase transition near p = 0.5—a very steep shift from "almost all correct" to "almost all wrong." Like water freezing at 0°C: before the critical point, LoRA holds memories reliably; past it, memorization collapses rapidly.
3. Roughly 1 bit of memory per parameter
Estimates suggest LoRA's memory efficiency is about 1 bit per parameter. A rank-64 LoRA (a few million parameters) can memorize roughly a few thousand facts. That sounds small, but given that LoRA modifies only a tiny fraction of the model's parameters, the efficiency is quite impressive.
---
4. Why It Matters
Practical guidance
Choosing a LoRA rank used to be pure guesswork: "Is rank-8 enough? Try rank-16?" Now there's a formula: given the number of facts you need to store, you can derive the minimum required rank. Need to memorize 500 new facts? Just calculate—no more guessing.
Understanding LoRA itself
LoRA is often treated as a black box—we know it works, but not why. The memory law reveals its nature from an information-theoretic perspective: LoRA is a finite-capacity parametric memory. Increasing rank increases storage; once full, memorization fails.
Comparison with RAG
RAG stores knowledge in external databases—nearly unlimited capacity, but with retrieval overhead. LoRA compresses knowledge into parameters—limited capacity, but zero retrieval latency. The memory law quantifies exactly how "limited" that capacity is, giving concrete numbers to the trade-off between the two approaches.
---
5. Honest Assessment: Limitations and Open Questions
1. Validated only on 8B models: does the power law hold for larger models (70B, 405B)? The authors admit this hasn't been tested. 2. Greedy decoding limitation: the phase transition at p = 0.5 was observed under greedy decoding. With stochastic methods like nucleus sampling, the transition may be less sharp. 3. Memorization ≠ understanding: LoRA can memorize "the capital of Paris is Lima" without understanding why. Memory capacity and reasoning ability are different things. 4. Open-domain generalization: the experiments used structured triple memorization. Whether open-domain text follows the same law remains unclear.
---
6. One-Sentence Takeaway
LoRA is not a magic bottomless pocket—it is a parametric memory with a precise capacity ceiling, scaling linearly with parameter count, and overflowing once full. Knowing where the ceiling is beats blindly piling on parameters.
---
Paper: https://arxiv.org/abs/2605.30260 Code: https://github.com/zjunlp/ParametricMemoryLaw