MixSD: Self-Distillation Cuts Catastrophic Forgetting in LLM Knowledge Injection by Up to 75%

> Supervised fine-tuning (SFT) is the standard way to inject new knowledge into LLMs—new facts, functions, domain expertise. But the side effect is severe: models learn new knowledge while forgetting old capabilities. A team from CMU and the University of Toronto proposes MixSD, built on a surprising insight: forgetting is not caused by *too much* parameter change, but by the *wrong direction*—external target sequences mismatch the model's own generation distribution, forcing it to imitate low-probability tokens. MixSD's fix is self-distillation: generate supervision targets from the model itself, mixing "expert outputs with new knowledge" and "naive outputs without it," so the model learns new knowledge without drifting from its native distribution. On Qwen3-1.7B, average capability retention after SFT is just 14.3%; MixSD raises it to 39.9% (2.8×). On Qwen3-4B-It, retention goes from 43.2% (SFT) to 79.9%. Paper: arXiv:2605.16865.

• Title: MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
• Authors: Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan, Zhijing Jin, Mona T. Diab
• Institutions: CMU, University of Toronto (Jinesis Lab & Vector Institute), UIUC, Princeton, Cornell, University of Tokyo (RIKEN AIP), Max Planck Institute for Intelligent Systems
• Code: https://github.com/jiarui-liu/MixSD

Key points

1. Root cause: direction, not magnitude of updates

• The popular explanation—forgetting comes from excessive parameter updates (motivating regularizers like EWC, LwF)—is contradicted by experiments (Sec. 7.1, Tables 5–6):

| Model | Correlation of ‖Δθ‖ with forgetting | Correlation of Fisher alignment ratio with forgetting | |---|---|---| | Qwen3-1.7B | r = +0.34 | r = +0.56 | | Qwen3-4B-It | r = +0.02 | r = +0.82 | | Qwen3-8B | r = +0.10 | r = +0.57 |

• Striking example (Qwen3-4B-It, KGF ACT-SMALL): SFT has ‖Δθ‖₂ = 28.2 with 39.4% forgetting; MixSD (λ=0.3) has a *larger* ‖Δθ‖₂ = 63.5 but only 8.4% forgetting.
• The true culprit: SFT targets written externally contain low-probability styles, formats, and reasoning structures. Fitting them pushes updates along Fisher-sensitive directions (alignment ratio R≈0.99 for SFT vs. 0.65–0.70 for MixSD).

2. Method: two conditions, one mix

• Expert-conditional output (context contains the real facts) carries the new knowledge signal.
• Naive-conditional output (no knowledge context) anchors the model to its original distribution.
• Per-token Bernoulli mixing: y_t^mix = ỹ+ with prob (1−λ), ỹ− with prob λ; training loss is standard NLL on the mixed sequence.
• λ controls the retention/memory tradeoff: λ=0 ≈ self-generated SFT; λ=0.3 recommended; λ=0.5–0.7 maximize retention at some memory cost.
• No external teacher model is required.

3. Headline results

KGF ACT-SMALL, Qwen3-1.7B (average retention across AIME-2024, MATH-500, GSM8K, HumanEval, MMLU):

| Method | Train acc. | Avg. retention | |---|---|---| | SFT | 99% | 14.3% | | OPSD | 99% | 5.1% | | MixSD (λ=0.3) | 100% | 39.9% | | MixSD (λ=0.5) | 97% | 40.3% (79% retrieval recall) |

KGF ACT-SMALL, Qwen3-4B-It: SFT retains 43.2%; MixSD (λ=0.7) retains 79.9%, near the base model's 82.6%.

Other highlights:

• KGF UNC (unseen function generalization): SFT collapses on unseen operations (Qwen3-4B-It: 1.4%); MixSD (λ=0.5) reaches 79.0%.
• KGF ACT-LARGE (Qwen3-4B-It): SFT retains 39.2%; MixSD retains 68.5% with 98.7% retrieval recall.
• SimpleQA (Qwen3-4B-It): SFT 26.6% → MixSD (λ=0.3) 76.2%.
• MQuAKE knowledge editing: MixSD matches SFT edit accuracy while retaining >90% of original capabilities.
• Llama-3.2-1B confirms the effect is architecture-agnostic.

4. Token-level NLL analysis (why it works)

• Fraction of supervision tokens with NLL > 8 (probability < 0.034%): SFT has 27–42%, MixSD only 0.2–5.7%. Aligning targets with the model's distribution is the direct mechanism.

5. Deployment implications

• Drop-in replacement for standard SFT; no architecture changes; no external teacher.
• Tunable λ trades off memory vs. retention per scenario.
• The Fisher alignment ratio offers a new diagnostic: regularization methods that merely *limit update magnitude* (EWC, LwF, SI) may treat symptoms rather than the cause.

6. Limitations

• λ must be tuned manually; no automatic selection.
• Requires two generations per sample (expert and naive conditions), so slower than SFT.
• Same-distribution function learning (KGF UNC-TEST) trails OPSD.
• Solves forgetting only; it does not improve learning efficiency of new knowledge.
• Tested only at 1.7B–8B scale.

Bottom line

MixSD's core claim is counterintuitive: catastrophic forgetting stems not from how far parameters move but *where* they move. By generating supervision from the model's own distribution—mixing expert and naive conditional outputs—MixSD keeps updates away from Fisher-sensitive directions. The result: near-baseline capability retention (79.9% on Qwen3-4B-It vs. 82.6% base) while still learning new knowledge. For any SFT-based knowledge injection scenario—RAG alternatives, domain adaptation, knowledge editing—MixSD offers a directly usable upgrade, and its Fisher alignment analysis suggests the field should shift from "limiting updates" to "steering directions."

References

• Paper: arXiv:2605.16865 (MixSD: Mixed Contextual Self-Distillation for Knowledge Injection)
• Code: https://github.com/jiarui-liu/MixSD
• Evaluation: KGF ACT/UNC/LARGE, SimpleQA, MQuAKE, AIME-2024, MATH-500, GSM8K, HumanEval, MMLU