The Squeezing Effect: A Deep-Dive Verification Report on Catastrophic Forgetting in LLM Fine-tuning

Summary

Catastrophic forgetting — where an LLM learns new skills (e.g., politeness, coding) but loses pre-existing knowledge (e.g., answering basic physics questions) — is a persistent pain point in RLHF alignment and domain-specific SFT. This report examines the "Squeezing Effect in LLM Fine-tuning" research (https://www.alphaxiv.org/overview/2407.10490v4), which argues that forgetting is not knowledge being deleted, but knowledge being squeezed and deformed within the model's high-dimensional representation manifold.

Background

• Catastrophic forgetting means a model's performance on prior tasks drops sharply when learning new tasks. In LLM fine-tuning, this appears as a zero-sum game in parameter space: newly injected knowledge comes at the cost of old knowledge.
• The referenced study reframes forgetting geometrically: fine-tuning exerts force on the model's knowledge manifold, deforming rather than erasing it.

Methods and Key Findings

1. Unified learning-dynamics framework

The study uses step-wise decomposition: the impact of learning a single training sample is broken down into its cumulative, step-by-step effect on predictions of other samples, covering SFT, DPO, and GRPO under one framework.

2. Negative gradients: a double-edged sword in preference optimization

• In DPO, negative gradients on dispreferred outputs do not simply delete unwanted behavior.
• They tend to lower the confidence of all outputs, including the positive samples themselves, sometimes squeezing preferred outputs below the confidence of the base model's favorite answer.
• This explains the well-known "over-optimization" failure: training DPO too long or too hard makes even desired outputs unlikely.
• The effect is general — it occurs whenever large negative gradients are applied to low-confidence outputs.

3. Attention head "defection" and functional re-routing

• During fine-tuning, attention heads that previously handled logical reasoning and factual retrieval can be re-purposed to monitor politeness or formatting.
• Experiments show roughly 15–23% of attention heads are severely disrupted, with damage concentrated in the lower layers responsible for basic language understanding and knowledge retrieval — explaining degraded general ability after fine-tuning.

4. Explanations for familiar phenomena

• Over-optimization: accumulated large negative gradients compress confidence across the output distribution.
• Hallucination reinforcement: cross-sample gradient interference mixes knowledge across tasks, producing seemingly related but incorrect content.
• SFT's indirect boosts: SFT uses only positive samples; via gradient similarity it can *raise* confidence in unrelated outputs. But this gain is fragile and collapses once negative gradients (e.g., DPO) are introduced.

Mitigation Strategies

1. Avoid brute-force gradient pressure: use early stopping and dynamic learning-rate schedules to limit manifold deformation. 2. Manifold protectors:

• *Regularization*: EWC penalizes updates to parameters important for old tasks; knowledge distillation keeps the fine-tuned model close to its pre-tuning self.
• *Orthogonal updates*: O-LoRA constrains task subspaces to be orthogonal; OLieRA adds Lie-group constraints with multiplicative updates, preserving the parameter geometry and achieving state-of-the-art continual-learning results.

3. Gentler preference optimization: e.g., VPO (Value-based Preference Optimization) selectively dampens negative gradients when the model already lacks confidence in a negative sample; dynamic tuning of DPO's β parameter can also balance alignment vs. retention. 4. Model and data strategies: modular architectures like Mixture-of-Experts isolate new-task updates; data replay reminds the model of old knowledge; balanced data mixes during safety alignment prevent sacrificing knowledge for politeness.

Conclusion

Fine-tuning is never loss-free knowledge injection: it resembles a localized earthquake on the model's fragile high-dimensional neural topology. The squeezing effect, however, is manageable — through gentle training schedules, regularization, orthogonal updates, and smarter preference optimization, practitioners can gain targeted capabilities without turning the model into a "digital idiot" that only recites rehearsed scripts. No facts beyond the referenced study were added in this translation.