A Frustrating Finetuning Story
You spend a week carefully preparing 500 biomedical knowledge triples — "MRE11 interacts with ATRX", "BRCA1 regulates DNA repair pathways" — and feed them to a 7B model via LoRA finetuning.
After training, you test:
> Q: Which protein does MRE11 interact with? > A: ATRX. ✓
Perfect. Memorization accuracy: 99.6%.
Then you ask a slightly harder question:
> Q: Which protein interacts with both ATRX and MRE11 and is involved in DNA repair?
The model rambles. Both facts needed for this reasoning chain are in the training set. The model "knows" them — why can't it "use" them?
This is a formally named phenomenon in LLM finetuning: the Knowing-Using Gap.
Two Dimensions of "Knowing But Not Using"
Lu Dai et al. from HKUST(GZ), in the paper *Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning*, decompose the gap into two quantifiable dimensions:
- Accuracy Gap: at the end of training, memorization accuracy approaches 100%, but multi-step generalization accuracy is only 7%–18%. Six models, two domains, no exceptions.
- Temporal Lag: memorization saturates by epoch 2–10, but generalization only emerges 5–9 epochs later — if it emerges at all.
- Ground floor archive (early layers): new documents get stored here first
- Second-floor meeting room (middle layers): real reasoning happens here
- Top-floor warehouse (late layers): aggregation before output
- Before training: the map is all blue — no knowledge exists to move.
- When memorization saturates: clear off-diagonal red regions appear. Knowledge exists in early/late layers and *works* if manually moved to middle layers — but the diagonal is still blue: the model fails naturally.
- Continued training: red regions migrate toward the diagonal. In successful runs, knowledge eventually reaches the reasoning layers and generalization appears. In failed runs, the red stalls mid-way — knowledge is stuck.
- Token-position ablation: patching at entity positions works (0.64); at
<BOS>it doesn't (0.05). The effect genuinely comes from knowledge representations. - CoT prompting ablation: chain-of-thought helps somewhat (chaining 0.12 → 0.29) but far less than self-patching (0.44), and it *hurts* intersection tasks.
- Irrelevant-knowledge patching: patching unrelated facts performs far worse — it's not just any activation perturbation.
- Cross-context consistency: effective layer-pair patterns are consistent across prompts, confirming genuine knowledge transfer rather than prompt-specific noise.
- Title: Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
- Authors: Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong
- Institutions: HKUST(GZ), HKUST
- arXiv: 2607.08393
- Code: https://anonymous.4open.science/r/Mem2Gen-71FF
- Datasets: STaRK-Prime (biomedical), STaRK-MAG (academic)
Formally, with \(A_{\text{mem}}(t)\) and \(A_{\text{gen}}(t)\) as memorization and generalization accuracies, the gap is \(\Delta A = A_{\text{mem}}(T) - A_{\text{gen}}(T)\) and the lag is \(\Delta T = T_{\text{gen}} - T_{\text{mem}}\).
The data is unforgiving: on Qwen-2.5-7B with LoRA on the chaining task, \(T_{\text{mem}} = 10.4\), \(T_{\text{gen}} = 15.0\), \(\Delta T = 4.6\), and final generalization accuracy is only 0.303. Worse: scaling from 1.5B to 7B barely shrinks the lag, and injecting more knowledge *widens* the accuracy gap.
Self-Patching: A "Knowledge Relocation Experiment"
To find where knowledge gets stuck, the paper introduces self-patching, a causal intervention technique. Think of a three-floor office building:
During finetuning, new knowledge lands in the archive and warehouse. But the meeting room never sees the documents, so reasoning uses stale information.
Self-patching takes the anchor-token representation from a source layer \(l_s\) and copies it to the same anchor position at a target layer \(l_t\) in another prompt, then continues the forward pass and measures the change in the probability of the correct answer:
If \(\Delta I > 0\), layer \(l_s\) holds useful knowledge that was simply never routed to layer \(l_t\).
The Percolation Map: A Time-Lapse of Knowledge Diffusion
The paper's most striking experiment saves per-epoch checkpoints and runs self-patching scans on each, producing a time-lapse movie of knowledge percolation:
Why? Once memorization saturates, loss approaches zero, gradients vanish, and the model has no pressure left to update its internal routing.
The Knowledge-Circuit Misalignment Hypothesis
The paper's hypothesis:
> Finetuning first encodes new facts in easily-fitted "storage layers" (early or late), which support direct recall but are not reliably routed to the "computation layers" (middle) needed for multi-step reasoning. This spatial misalignment between knowledge storage and reasoning computation causes generalization failure.
Two testable predictions — both confirmed:
1. After memorization saturates, off-path knowledge representations already exist internally. 2. Manually patching them into position immediately improves generalization.
The second result is dramatic. On Qwen-2.5-7B, chaining accuracy jumps from 0.124 to 0.504 — a 4x improvement, *after* natural finetuning has converged. On LLaMA-3.2-1B, from 0.102 to 0.316 — 3x. Intersection-task gaps are nearly eliminated (from ~0.8 to 0.97+).
Ruling Out Alternative Explanations
From Diagnosis to Practice: Two Fixed Layer Pairs Recover 58–75%
Oracle self-patching requires an exhaustive per-instance layer-pair search. But effective patches cluster into two groups: late → middle (~\(0.8L \to 0.5L\)) and early → middle (~\(0.1L \to 0.5L\)).
A simple fixed heuristic — two preset layer pairs per architecture — recovers 58%–75% of the oracle headroom:
| Model | No intervention (chaining) | Fixed heuristic | Oracle | |------|:---:|:---:|:---:| | Qwen-2.5-1.5B | 0.078 | 0.349 | 0.440 | | Qwen-2.5-3B | 0.114 | 0.435 | 0.542 | | Qwen-2.5-7B | 0.124 | 0.409 | 0.504 | | LLaMA-3.2-1B | 0.102 | 0.252 | 0.316 | | LLaMA-3.2-3B | 0.126 | 0.321 | 0.404 | | LLaMA-3.1-8B | 0.182 | 0.375 | 0.458 |
Average chaining accuracy rises from 0.121 to 0.357 — a 3x gain from just two fixed layer-pair transports.
What This Means for Practitioners
1. Finetuning is not the endpoint — alignment is. Finetuning only stores knowledge; you must also ensure it reaches the reasoning path. Storage ≠ routing. 2. "Not enough capacity" may be a misdiagnosis. Going from 1.5B to 7B doesn't close the gap. The problem is routing, not capacity. 3. The loss function lies. Near-zero training loss means "memorized", not "learned". Gradient vanishing after saturation is a structural trap. 4. Middle layers are the reasoning "meeting room". This aligns with prior research locating multi-step reasoning in LLM middle layers. 5. Alignment-aware training is next. Future finetuning should explicitly optimize knowledge-to-reasoning-layer routing — e.g., via routing regularization or dedicated middle-layer alignment objectives.
A Deeper Analogy: The Library Reading Room
Imagine you're a librarian with a thousand new books. You stack them all in the basement (early layers) and the attic (late layers). Patrons can find titles and recite catalogs — that's memorization. But a researcher who needs to cross-reference three books needs them on the reading-room table (middle layers). The books exist; they're just not where the analysis happens.
Self-patching is the experiment of carrying books from the stacks to the reading room. If the analysis then succeeds, the content was fine — the problem was logistics. The Knowing-Using Gap is, at its core, a logistics problem.