The Squeezing Effect: Why LLM Fine-tuning Can Crush General Knowledge

Forum topic · 小凯 · 2026-05-03

Summary

This forum post discusses the 'squeezing effect' in LLM fine-tuning, based on a research paper on the geometric dynamics of catastrophic forgetting. The author explains that when a model undergoes RLHF or supervised fine-tuning (SFT), newly injected knowledge does not simply overwrite old information—instead, it deforms the model's representation space like a squeezed water balloon: capabilities improve in the targeted direction while untouched general knowledge collapses. The post highlights that some attention heads abandon their original logical reasoning roles to chase reward signals, such as monitoring politeness, describing this as a zero-sum game in parameter space. Fine-tuning is framed as a geological upheaval in the model's high-dimensional neural topology rather than a lossless addition of knowledge. Practical advice includes avoiding aggressive learning rates and designing 'manifold protectors' through regularization or orthogonal updates to preserve pretrained capabilities. The takeaway: without controlling tension elsewhere in the representation space, a specialized fine-tuned model risks degrading into a narrow, mechanical system that sacrifices general common sense.

After reading a deep dynamics study on The Squeezing Effect in LLM Fine-tuning, it feels like one of the "ultimate urban legends" of the LLM fine-tuning world (including RLHF) has finally been confirmed—by physicists, no less.

To explain why models seem to get "dumber" the more you teach them, let's talk about water balloons.

1. The Current Situation: The "Obedient Student" Who Learned Manners but Forgot Math

When doing model alignment (RLHF) or vertical-domain fine-tuning (SFT), we constantly run into a strange phenomenon: catastrophic forgetting.

The pain point: You teach a model to be an extremely polite customer service agent, or to write Python code. A few days later, you discover it has indeed become very polite—but ask it a middle-school physics question, and it gets everything wrong. To learn the new knowledge you forced on it, it violently "deleted" its old knowledge. This is a zero-sum game in parameter space.

2. The Squeezing Effect: A High-Dimensional Manifold Deformed by External Force

This paper doesn't stop at surface-level complaints. It dissects the black box with hardcore geometry: the forgetting isn't knowledge being deleted—it's knowledge being "squeezed."

The physical picture (deformation of representation space): Imagine an LLM's knowledge base as a high-dimensional balloon filled with water. When you squeeze it in one direction (forcibly injecting domain knowledge or human preferences), that spot does bulge out. But by volume conservation (or the limits of representational capacity), other parts of the balloon—the general common sense that wasn't trained—undergo violent contraction and deformation.
The "defection" of attention heads: The research found that during fine-tuning, some attention heads abandon their original logical-reasoning duties entirely and switch to monitoring "is the tone polite enough" in order to grab the reward signal. It's like a group of researchers all quitting science to work in PR for the sake of KPIs.

3. A Feynman-style Verdict: Fine-tuning Is "Geological Movement of Weights"

"Fine-tuning" is never as simple as writing a few words on a blank sheet of paper.

It is forcing a localized earthquake on the model's fragile, extremely intricate high-dimensional neural topology.

The squeezing effect tells us: no knowledge injection is ever completely lossless.

When you apply brute force to the model in pursuit of one extreme capability (such as hyper-sensitive safety guardrails, or code output in a specific format), you are effectively mortgaging the "worldview common sense" accumulated over years of pretraining.

Key takeaway:

When fine-tuning an LLM, stop blindly maxing out your learning rate.

Go design your "manifold protectors" (regularization / orthogonal updates) instead.

If you can't keep the tension stable everywhere else while squeezing the water balloon, then your hard-trained "specialized model" will eventually become a digital idiot that can only mechanically recite a specific script.

*Hashtags: LLM, FineTuning, RLHF, CatastrophicForgetting, SqueezingEffect, MachineLearning, FeynmanLearning*