MixSD: Self-Distillation for Knowledge Injection That Cuts Catastrophic Forgetting in LLM Fine-Tuning
Supervised fine-tuning (SFT) is the standard way to inject new knowledge into large language models — new facts, new functions, new domain material. But the side effect is disastrous: the model learns the new knowledge and forgets much of what it previously knew. A team from CMU and the University of Toronto proposes MixSD, built on a surprising insight: forgetting is not caused by how much the parameters change, but by the direction of the update. Externally written target sequences mismatch the model's own generation distribution, forcing it to imitate low-probability tokens. MixSD's fix is self-distillation: use the model itself to generate supervision targets, mixing an "expert output with new knowledge" and an "original output without new knowledge," so the model learns new facts without drifting off its native distribution. On Qwen3-1.7B, SFT left only 14.3% of average capability retained, while MixSD raised this to 39.9% (2.8x). On Qwen3-4B-It, SFT retained 43.2% versus MixSD's 79.9%. Paper: arXiv:2605.16865.
Key points
- Root cause of forgetting is direction, not magnitude. On Qwen3-4B-It (KGF ACT-SMALL): SFT had ‖Δθ‖₂ = 28.2 with 39.4% forgetting, while MixSD (λ=0.3) had a *larger* displacement of 63.5 with only 8.4% forgetting. Correlation between parameter displacement and forgetting is weak (r = 0.02–0.34), but Fisher alignment correlates strongly (r = 0.56–0.82).
- Method. For each input, the model generates two candidate outputs:
- *Expert conditional* (with true facts injected in context) — carries the new knowledge signal;
- *Naive conditional* (no injected knowledge) — anchors the model to its original distribution. Each token is then sampled from the expert stream with probability 1−λ or the naive stream with probability λ (λ = 0.3 recommended). Training uses standard negative log-likelihood on this mixed sequence.
- Fisher alignment ratio (R = (Δθᵀ F̂ Δθ / ‖Δθ‖²) / (tr(F̂)/d)) quantifies whether updates concentrate on sensitive directions. SFT: R ≈ 0.99; MixSD: R = 0.65–0.70. This suggests regularization approaches that limit update magnitude (EWC, LwF) may treat the symptom, not the cause.
- Token-level NLL analysis. In SFT targets, 27–42% of tokens have NLL > 8 (probability < 0.034%); MixSD reduces this to 0.2–5.7%.
- λ requires per-scenario tuning; no automatic selection mechanism
- Requires two forward passes per example, slower than SFT
- In-distribution generalization on KGF UNC-TEST trails OPSD
- Addresses forgetting only, not learning efficiency
- Tested only at 1.7B–8B scale
- Paper: MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
- Authors: Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan, Zhijing Jin, Mona T. Diab
- Institutions: CMU, University of Toronto, UIUC, Princeton, Cornell, University of Tokyo (RIKEN AIP), Max Planck Institute for Intelligent Systems
- arXiv: 2605.16865
- Code: https://github.com/jiarui-liu/MixSD
Main results
KGF ACT-SMALL (Qwen3-1.7B):
| Method | Train acc | Avg retention | |---|---|---| | Base | – | 56.5% | | SFT | 99% | 14.3% | | OPSD | 99% | 5.1% | | MixSD (λ=0.3) | 100% | 39.9% | | MixSD (λ=0.5) | 97% | 40.3% |
Qwen3-4B-It: SFT retained 43.2%; MixSD (λ=0.7) retained 79.9%, near the base model's 82.6%.
KGF UNC (unseen function generalization, Qwen3-4B-It): SFT collapsed to 1.4% on unseen operations; MixSD (λ=0.5) reached 79.0%.
KGF ACT-LARGE: Qwen3-1.7B SFT retention dropped to 0.5% vs MixSD's 20.7%; Qwen3-4B-It MixSD (λ=0) retained 68.5% with 98.7% retrieval recall.
SimpleQA (Qwen3-4B-It): SFT retained 26.6%; MixSD (λ=0.3) retained 76.2%.
MQuAKE (knowledge editing): MixSD (λ=0.3) achieved comparable edit accuracy to SFT while retaining >90% of existing capabilities (SFT: 7.8–39.4%).
Cross-family validation (Llama-3.2-1B): SFT retention 1.4 vs MixSD (λ=0.5) 5.3 — the effect is architecture-independent.
Deployment implications
1. Drop-in replacement for standard SFT with no architecture changes 2. No external teacher model required 3. Tunable λ balances memorization vs retention 4. Generalizes across model families (Qwen, Llama)
Limitations
Conclusion
MixSD's core insight is counterintuitive: catastrophic forgetting comes from updating in the wrong direction, driven by distribution mismatch between external targets and the model's own distribution. By generating supervision targets from the model itself — mixing knowledge-bearing expert output with naive output — MixSD aligns supervision with the model's distribution and preserves capabilities close to base-model levels. The introduced Fisher alignment ratio further suggests future anti-forgetting research should shift from "limiting update magnitude" to "guiding update direction," making MixSD a practical upgrade for any SFT-based knowledge injection scenario (domain adaptation, knowledge editing, alternatives to RAG).
References