A Frustrating Fine-Tuning Story
You spend a week carefully preparing 500 biomedical knowledge triples—"MRE11 protein interacts with ATRX protein", "BRCA1 gene regulates the DNA repair pathway"—and feed them to a 7B-parameter LLM for LoRA fine-tuning.
After training, you test with high expectations:
> Q: Which protein does MRE11 interact with? > A: ATRX. ✓
Perfect. The model memorized everything, with 99.6% accuracy.
Then you ask a slightly more complex question:
> Q: Which protein interacts with both ATRX and MRE11, and is involved in DNA repair?
The model is lost and starts hallucinating.
You check the training data—both facts needed for the reasoning (MRE11–ATRX interaction, ATRX–DNA repair association) are in the training set. The model clearly "knows" these facts. Why can't it "use" them?
This isn't your fault, and it isn't a model bug. It's a formally named phenomenon in LLM fine-tuning—the Knowing-Using Gap.
Two Dimensions of "Knowing but Not Using"
Lu Dai et al. from HKUST(GZ), in the paper *Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning*, decompose the phenomenon into two quantifiable dimensions:
Accuracy Gap: At the end of training, memorization-test accuracy approaches 100%, but generalization tests requiring multi-step reasoning reach only 7%–18%. Across six models and two domains, without exception.
Temporal Lag: Memorization saturates within 2–10 epochs, but generalization only begins 5–9 epochs later—if it appears at all. Often, it never does.
The paper gives a precise mathematical definition. Let \(A_{\text{mem}}(t)\) be memorization accuracy at time \(t\) and \(A_{\text{gen}}(t)\) generalization accuracy. The accuracy gap is \(\Delta A = A_{\text{mem}}(T) - A_{\text{gen}}(T)\), and the temporal lag is \(\Delta T = T_{\text{gen}} - T_{\text{mem}}\).
The data is cold: on Qwen-2.5-7B with LoRA fine-tuning on chaining tasks, \(T_{\text{mem}} = 10.4\) epochs, \(T_{\text{gen}} = 15.0\) epochs, \(\Delta T = 4.6\). And the final generalization accuracy is only 0.303—even when generalization eventually appears, the model gets only 30% of answers right.
More painfully: scaling up the model doesn't solve the problem. From 1.5B to 7B, the temporal lag barely shrinks. Increasing the amount of injected knowledge actually widens the accuracy gap—storing more, using worse.
Self-Patching: A "Knowledge Relocation Experiment"
To figure out where knowledge gets stuck, the paper proposes a causal intervention technique called self-patching.
First, an analogy. Imagine a three-story office building:
- Ground-floor archive (early model layers): new documents arrive and are filed here first
- Second-floor meeting room (middle layers): real reasoning and decision-making happen here
- Top-floor warehouse (late layers): aggregation and final output preparation
- Late → middle layers (~\(0.8L \to 0.5L\))
- Early → middle layers (~\(0.1L \to 0.5L\))
- Title: Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
- Authors: Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong
- Affiliations: HKUST(GZ), HKUST
- arXiv: 2607.08393
- Code: https://anonymous.4open.science/r/Mem2Gen-71FF
- Datasets: STaRK-Prime (biomedical), STaRK-MAG (academic)
During fine-tuning, new knowledge is quickly stored in the archive and the warehouse. But the people in the meeting room don't know the document exists, so they reason with outdated information—hence "knowing but not using".
The self-patching experiment is brutally simple: take a document from the archive (source layer \(l_s\)), manually insert it into the meeting room (target layer \(l_t\)), and see if reasoning improves. Then enumerate all layer combinations \((l_s, l_t)\) and draw a "percolation map".
Concretely, for the hidden state \(h_t^l(P)\) at layer \(l\), token position \(t\) of model \(M\), self-patching copies the anchor representation of source prompt \(P_s\) at source layer \(l_s\) into the same anchor position of target prompt \(P_t\) at target layer \(l_t\), continues the forward pass, and measures the change in the probability of the correct answer:
If \(\Delta I > 0\), the knowledge representation exists at layer \(l_s\) but simply wasn't routed to layer \(l_t\) to participate in computation.
The Percolation Map: A Movie of Knowledge Diffusion
The paper's most beautiful experiment saves checkpoints epoch by epoch and runs self-patching scans on each, producing a time-lapse movie of "knowledge percolation dynamics".
Before training: the map is all blue. Knowledge hasn't entered the model; relocating anything does nothing.
When memorization saturates: clear off-diagonal red regions appear. Knowledge already exists in certain layers (typically early or late), and manually moving it to the middle layers makes reasoning succeed. But the diagonal itself is still blue—the model in its natural state still answers incorrectly.
Continuing training: the red regions gradually spread toward the diagonal. In successful cases, red eventually covers the diagonal—knowledge naturally percolates into the reasoning layers, and generalization appears. In failed cases, the red regions stall and never reach the diagonal—knowledge is stuck halfway.
Why does it get stuck? Because once memorization saturates, the loss approaches zero, gradients vanish, and the model no longer has any incentive to update its internal representations. Knowledge remains permanently in a "stored but unusable" position.
The Knowledge-Circuit Misalignment Hypothesis: A Routing Problem, Not a Capacity Problem
Based on these observations, the paper proposes the Knowledge-Circuit Misalignment Hypothesis:
> Fine-tuning first encodes new facts in easily-fitted "storage layers" (typically early or late layers), which support direct recall but are not reliably routed to the "computational layers" (typically middle layers) required for multi-step reasoning. This spatial misalignment between knowledge storage and reasoning computation causes generalization failure.
The hypothesis makes two testable predictions:
1. After memorization saturates, "off-path" knowledge representations should already exist inside the model—they contain the information but cannot produce correct end-to-end reasoning. ✓ Confirmed. 2. Manually moving these representations to the right place should immediately improve generalization. ✓ Confirmed.
The verification of the second prediction is stunning. On Qwen-2.5-7B, chaining accuracy jumps from 0.124 to 0.504—a 4x improvement, even after natural fine-tuning has converged. On LLaMA-3.2-1B, from 0.102 to 0.316—a 3x improvement.
Across six models and two domains (biomedical STaRK-Prime and academic STaRK-MAG), the results are highly consistent. The intersection-task gap is nearly eliminated (from ~0.8 up to 0.97+).
Ruling Out Alternative Explanations
The paper runs key ablations to rule out alternative explanations like "the patching itself has a distracting effect":
Token position ablation: Patching at entity positions is most effective (0.64); patching at the <BOS> position is ineffective (0.05). The effect genuinely comes from knowledge representations, not arbitrary perturbations.
CoT prompting ablation: Chain-of-Thought prompting helps somewhat (chaining from 0.12 to 0.29), but far less than self-patching (0.44). CoT even hurts performance on intersection tasks.
Irrelevant knowledge patching: Patching the representation of an unrelated fact performs far worse than patching the correct fact's representation. This rules out "any activation perturbation helps".
Cross-context consistency: Patching from memorization-prompt representations to generalization prompts yields highly consistent effective layer-pair patterns. What is being patched is the knowledge representation itself, not prompt-specific noise.
From Diagnosis to Practice: Two Fixed Layer Pairs Recover 58–75% of the Gap
Oracle self-patching requires exhaustive layer-pair search per instance—far too expensive. But the paper finds a pattern: effective patches cluster in two groups:
Based on this, the paper designs an extremely simple fixed heuristic: just two preset layer pairs per model architecture, no per-instance search.
Result: across six models, the fixed strategy recovers 58%–75% of the oracle headroom.
| Model | No intervention (Chaining) | Fixed strategy | Oracle | |------|:---:|:---:|:---:| | Qwen-2.5-1.5B | 0.078 | 0.349 | 0.440 | | Qwen-2.5-3B | 0.114 | 0.435 | 0.542 | | Qwen-2.5-7B | 0.124 | 0.409 | 0.504 | | LLaMA-3.2-1B | 0.102 | 0.252 | 0.316 | | LLaMA-3.2-3B | 0.126 | 0.321 | 0.404 | | LLaMA-3.1-8B | 0.182 | 0.375 | 0.458 |
On average, chaining accuracy improves from 0.121 to 0.357—a 3x gain—from just two fixed inter-layer relocations.
What This Means for AI Practitioners
1. Fine-tuning is not the endpoint; alignment is. The traditional view holds "fine-tuning = injecting knowledge". This paper says: fine-tuning only puts knowledge into the warehouse—you still need to make sure knowledge reaches the reasoning path. Storage ≠ routing.
2. "Not enough model capacity" may be a misdiagnosis. When your fine-tuned model performs poorly on reasoning tasks, your first instinct may be "the model is too small" or "not enough data". But scaling from 1.5B to 7B does not eliminate the gap. The problem is not capacity—it's routing.
3. The loss function lies. When training loss approaches zero, you think the model "learned". In reality, it merely "memorized". After memorization saturates, gradients vanish and the model loses the drive to adjust its internal routing. This is a structural trap.
4. Middle layers are the reasoning "meeting room". The paper aligns with prior research on middle-layer computation in LLMs: the key computations for multi-step reasoning happen in the middle layers. If your knowledge stays in early or late layers, it never reaches the meeting room.
5. Alignment-aware training is the next step. The paper argues future fine-tuning methods should explicitly optimize routing knowledge to reasoning layers, not just minimize memorization loss. That might mean adding "routing regularization" to the loss or designing dedicated middle-layer alignment objectives.
A Deeper Analogy: The Library and the Reading Room
One final analogy. Imagine you are a head librarian who just bought a thousand new books.
You stack them all in the underground vault (early layers) and the top-floor warehouse (late layers). Readers come asking; you tell them "the books are in the vault" or "in the warehouse". Readers can find titles and recite catalogs—this is "memorization".
But when a researcher needs to cross-reference three books simultaneously, they need the books moved to the third-floor reading room (middle layers), spread out on a table. If the books stay in the vault, cross-analysis is impossible—not because the books don't exist, but because they never arrived where they're needed.
Self-patching is the experiment: move books from the vault to the reading room and see whether cross-analysis succeeds. If it does, the content is fine—the problem is logistics.
The Knowing-Using Gap in fine-tuned models is, at its core, a logistics problem.