The Squeezing Effect: How Catastrophic Forgetting Works in LLM Fine-tuning

Forum topic · ✨步子哥 · 2026-05-03

Summary

This report examines the 'Squeezing Effect' in large language model (LLM) fine-tuning, a mechanism underlying catastrophic forgetting during alignment (RLHF, DPO, GRPO) and supervised fine-tuning (SFT). Rather than old knowledge being deleted, the study argues that fine-tuning 'squeezes' and deforms the model's knowledge manifold. Using step-wise learning dynamics decomposition, the research reveals two key mechanisms: (1) negative gradients in preference optimization suppress confidence in all outputs, including preferred ones, so over-trained DPO models can push desired outputs below their initial probabilities; and (2) attention heads get functionally 'redirected'—an estimated 15-23% of heads become severely disrupted, especially in lower layers responsible for basic language understanding and knowledge retrieval. The analysis also explains why over-optimization backfires, why fine-tuning can reinforce hallucinations via cross-sample gradient interference, and why SFT can indirectly raise confidence on unrelated outputs. The report proposes mitigation strategies: early stopping and learning-rate control, regularization (EWC, knowledge distillation), orthogonal update methods (O-LoRA, OLieRA), gentler preference optimization (VPO, dynamic beta tuning), modular architectures (MoE), and data replay. The conclusion: no knowledge injection is lossless, but careful training can achieve low-loss fine-tuning that preserves general capabilities.

The Squeezing Effect: A Deep-Dive Verification Report on Catastrophic Forgetting in LLM Fine-tuning

Abstract

Catastrophic forgetting during LLM alignment (RLHF) and vertical-domain fine-tuning (SFT) remains a persistent pain point: models learn new skills (politeness, coding) yet lose prior knowledge (basic physics). Often described as "getting dumber the more you teach it," this is essentially a zero-sum squeeze in parameter space. The research "The Squeezing Effect in LLM Fine-tuning" (https://www.alphaxiv.org/overview/2407.10490v4) dissects this black box from a geometric perspective, showing that forgetting is not knowledge deletion but knowledge being squeezed and deformed.

Background

Catastrophic forgetting is the classic problem where learning a new task sharply degrades performance on earlier tasks. In LLMs this is acute: broad pretraining knowledge degrades after task- or preference-specific fine-tuning.
Traditional views frame forgetting as "old knowledge overwritten or deleted." The Squeezing Effect study instead treats LLM knowledge representations as a high-dimensional manifold that fine-tuning deforms under external force.

Methods and Key Findings

1. A Unified Learning-Dynamics Framework

The study unifies SFT, DPO, and GRPO under one learning-dynamics lens via step-wise decomposition: the effect of learning a single training sample is decomposed into cumulative, step-by-step effects on predictions of other samples, tracking how confidence distributions over outputs evolve.

2. Mechanism 1: Negative Gradients in Preference Optimization

In DPO, models see positive (preferred) and negative (rejected) samples, maximizing their gap.
Negative gradients do not simply delete unwanted behavior. They tend to lower confidence in all outputs, including the positive samples themselves—the preferred output's absolute confidence can drop below even the base model's favored output.
This "squeezing-out" is a general phenomenon wherever large negative gradients hit low-confidence outputs.

3. Mechanism 2: Attention Head "Defection"

Attention heads responsible for logical reasoning and fact retrieval can be functionally redirected toward monitoring politeness or format compliance during fine-tuning.
Experiments show ~15–23% of attention heads are severely disrupted, with damage concentrated in lower layers responsible for basic language understanding and knowledge retrieval—explaining degraded general-task performance after fine-tuning.

4. Experimental Explanations

Over-optimization backfires: prolonged DPO training compresses confidence in all outputs, so even desired outputs become unlikely. Training duration and intensity must be carefully controlled.
Hallucination reinforcement: cross-sample gradient interference can entangle knowledge from different tasks, causing models to mix facts across contexts.
SFT's indirect boosts: SFT raises confidence in unrelated answers via gradient similarity—fragile gains that collapse once negative gradients (e.g., DPO) are introduced.

Mitigation Strategies

1. Avoid brute-force gradient pressure: use early stopping and dynamic learning-rate decay to reduce shocks to existing representations. 2. "Manifold protectors":

*Regularization*: Elastic Weight Consolidation (EWC), knowledge distillation from the pre-fine-tuned model, and instruction-vector frameworks to preserve specific computation paths.
*Orthogonal updates*: O-LoRA constrains task subspaces to be orthogonal in LoRA fine-tuning; OLieRA adds Lie-group constraints with multiplicative updates, preserving intrinsic parameter geometry and achieving state-of-the-art continual-learning results.

3. Gentler preference optimization: VPO (Value-based Preference Optimization) selectively attenuates negative gradients when the model is already unconfident about negative samples; dynamically tuning DPO's beta parameter also balances alignment against retention. 4. Model and data strategies: modular architectures such as Mixture-of-Experts isolate old and new capabilities; data replay interleaves old-task data during fine-tuning; diversified data mixes (e.g., pairing safety data with general QA) prevent over-sacrificing knowledge for safety.

Conclusion

Fine-tuning is not writing on a blank page—it is a localized earthquake on the model's fragile high-dimensional neural topology. Squeezing one part of the "knowledge balloon" inevitably deforms the rest. No knowledge injection is fully lossless, but through gentle training schedules, regularization, orthogonal updates, and smarter preference optimization, practitioners can obtain desired specialization while keeping general capabilities intact—moving toward truly low-loss fine-tuning.