The Squeezing Effect in LLM Fine-tuning: Why Feeding Knowledge Can Crush Common Sense

Forum topic · 小凯 · 2026-05-03

Summary

This forum post discusses the 'squeezing effect' observed in large language model (LLM) fine-tuning, including RLHF and supervised fine-tuning. The author explains catastrophic forgetting through a geometric analogy: fine-tuning deforms the model's representation space like squeezing a water balloon—knowledge is not deleted but compressed and displaced as new domain knowledge or human preferences are injected. The post highlights research showing that certain attention heads abandon logical reasoning tasks to pursue reward signals (e.g., politeness), and frames fine-tuning as a 'geological movement' of weights rather than innocuous additions. The author concludes that no knowledge injection is lossless and recommends regularization and orthogonal updates to protect the model's pretrained general capabilities, warning against excessive learning rates that trade general common sense for narrow specialization.

The Squeezing Effect in LLM Fine-tuning: Are You Feeding Knowledge or Crushing Common Sense?

After reading a deep dynamics study on The Squeezing Effect in LLM Fine-tuning, it feels like one of the "ultimate urban legends" of the LLM fine-tuning (including RLHF) world has finally been confirmed by physicists.

To explain why LLMs get "dumber" the more you teach them, let's talk about water balloons.

1. The Current Situation: The "Well-Behaved Student" Who Learned Politeness but Forgot Math

When doing LLM alignment (RLHF) or domain-specific fine-tuning (SFT), we often encounter a strange phenomenon: Catastrophic Forgetting.

The pain point: You teach a model to become an extremely polite customer service agent, or how to write Python code. A few days later, you find it is indeed very polite—but when you ask it a middle school physics question, it gets everything wrong. To learn the new knowledge you taught it, it forcibly "deleted" the old knowledge. This is a "zero-sum game in parameter space."

2. The Squeezing Effect: A High-Dimensional Manifold Deformed by External Force

The paper doesn't stop at surface-level complaints; it dissects the black box with hardcore geometry: this forgetting is not knowledge being deleted—it is knowledge being "squeezed."

The physical picture (deformation of representation space): Imagine an LLM's knowledge base as a high-dimensional balloon filled with water. When you squeeze it in one direction (forcibly injecting specific domain knowledge or human preferences), that part of the balloon does bulge out. But by volume conservation (or the limits of representational capacity), the other parts of the balloon—the untapped general common sense—undergo drastic contraction and deformation.
Attention heads "defecting": Research has found that during fine-tuning, some attention heads abandon their original logical reasoning duties in order to capture the reward signal, switching entirely to monitoring "whether the tone is polite enough." It's like a group of researchers all switching careers to PR just to hit their KPIs.

3. A Feynman-Style Judgment: Fine-tuning Is "Geological Movement of Weights"

So-called "fine-tuning" is never as simple as writing a few words on a blank sheet of paper.

It is forcibly triggering a localized earthquake on the model's fragile and extremely precise high-dimensional neural topology.

The squeezing effect tells us: no knowledge injection is completely lossless.

When you apply brute-force pressure on a model to pursue one extreme capability (such as hyper-sensitive safety guardrails, or code output in a specific format), you are effectively mortgaging the "worldview and common sense" accumulated over years of pretraining.

Key takeaway:

When fine-tuning an LLM, stop blindly maxing out your learning rate.

Design your "manifold protectors" (Regularization / Orthogonal Updates) instead.

If you can't keep the tension stable everywhere else while squeezing the water balloon, then the "specialized model" you worked so hard to train will ultimately become a digital idiot that can only mechanically recite a particular script.