Spurious Forgetting in LLMs: When Forgetting Reverses Itself — A Three-Act Play and a Three-Ingredient Mechanism

Forum topic · ✨步子哥 · 2026-10-07

Summary

A 2026 arXiv paper by Palit et al., 'When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting', dissects a surprising phenomenon in large language models: apparent catastrophic forgetting that reverses itself. In synthetic biography experiments, recall of old data first collapses to about 20% accuracy, then spontaneously recovers to 80% during fine-tuning on new data, before slowly eroding. The authors show this 'spurious forgetting' requires only three ingredients: shared structure among old keys, new answers concentrated in a localized region, and normalization that cancels a global shift. Fine-tuning pushes all old representations together; normalization makes them look forgotten, but the knowledge is merely hidden. Causal experiments confirm this: removing the shared offset direction from weight updates eliminates collapse while preserving new-task accuracy, and can even restore old-knowledge recall beyond the fine-tuned model. True forgetting corresponds only to the slow, irreversible 'erosion' phase. The findings imply that practitioners should not rush to replay old data on seeing capability drops, that evaluations must examine full learning curves, and that defenses against collapse (normalization/representation adjustments) differ from defenses against genuine erosion (replay). Paper: arXiv:2610.08718; code on GitHub.

Spurious Forgetting in LLMs: Forgetting That Undoes Itself — A Three-Act Play and a Three-Ingredient Mechanism

An Experimental Scenario

Train a language model to memorize the birthdays of 1,000 people. Then fine-tune it on the birthdays of 200 new people—without ever revisiting the old data. Common sense says the old memories should gradually decay. But in experiments something eerie happens: old-memory accuracy first plummets to ~20%, then—without you doing anything—recovers on its own to ~80%, and only afterward begins a slow decline.

This is not a bug, nor training noise. In October 2026, Palit et al. published the arXiv paper *When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting*, which for the first time precisely dissects the mechanism of this "spurious forgetting": forgetting does not erase knowledge—it merely hides it, and later training digs it back out.

The Three Acts: Collapse, Recovery, Erosion

The paper uses a minimal experiment with synthetic biographies. A Transformer is first pretrained on two disjoint biography sets A and B, then fine-tuned on new biographies without reviewing any old ones.

When the new biographies overlap with both A and B (set C), classic catastrophic forgetting appears: recall of A and B slowly and steadily declines.

But when the new biographies overlap only with B (set B'), something strange happens—B's recall is barely affected, while A's recall follows a three-phase trajectory:

1. Collapse: A's recall drops rapidly even before the new knowledge is learned. 2. Recovery: As the new knowledge is acquired, A's recall rises back on its own—even though A never reappears in training data. 3. Erosion: Only then does A's recall begin a slow, sustained decline.

In other words: the model "forgets" A, then "remembers" it again. What happens in between?

The Three Ingredients: Shared Structure, Localized Region, Normalization

The paper's core contribution is reproducing the phenomenon in a minimal associative memory model, showing that only three ingredients are needed:

Ingredient 1: Keys have shared structure. The representation keys of old knowledge share common components. When fine-tuning moves one key, all keys sharing that structure move together.

Ingredient 2: New answers are concentrated in a region. The answers of the new knowledge cluster in a localized area of representation space, so fine-tuning pushes that region in one direction.

Ingredient 3: Normalization. Softmax/normalization operations cancel out the global shift.

Together, the three ingredients mean: during fine-tuning, all old representations are pushed off together, and normalization makes them look "forgotten"—but the knowledge remains, hidden by a global offset. Once the new knowledge is learned, the push stops, normalization cancels the offset, and the old knowledge resurfaces.

That is the mechanism behind "collapse → recovery." The final "erosion" is true forgetting—per-item changes accumulating slowly and irreversibly.

Causal Evidence: Remove the Shared Offset and the Collapse Disappears

Beyond phenomena and models, the paper provides three causal experiments:

Experiment 1: In a from-scratch Transformer, removing the common offset in representations eliminates the collapse. The end of the collapse is determined by when the new knowledge is learned—not by training steps.

Experiment 2: In pretrained language models, old-knowledge recall only recovers when the new biographies' subjects do not leak the answers. If the new knowledge itself hints at the old knowledge, the mechanism breaks down.

Experiment 3: Most strikingly—removing a single direction from each weight update restores old-knowledge recall to a level *exceeding* the fine-tuned model, while keeping new-knowledge accuracy unchanged.

In other words, one direction in the weight updates is the culprit of the "shared offset." Remove it, and spurious forgetting disappears while true forgetting is unaffected.

What This Means

For training practice: If old capabilities crash during fine-tuning, don't rush to replay (replay old data). Keep training a few more rounds and see whether performance comes back on its own. If it's spurious forgetting, replay may actually interfere with genuine learning.

For model evaluation: Don't judge a fine-tuned model by its endpoint alone. The three-act structure means any single intermediate reading may be an artifact. You need the full learning curve.

For continual learning: True forgetting is the "erosion" phase, not the "collapse" phase. Distinguishing the two is essential for designing correct defenses: counter collapse by adjusting normalization or representation structure; counter erosion with replay.

A Cross-Domain Analogy: The Library's "Wholesale Relocation"

Imagine a library where every bookshelf is bolted to a floor that can slide as a whole. You add a batch of new books in one corner, and to make room, you nudge the entire floor in that direction.

Every old book's position shifts—readers using the old coordinates find nothing. That is "collapse."

But the library has a normalization mechanism: an indexing system auto-recalibrates, canceling the global offset. Once calibrated, the old books are findable again. That is "recovery."

True forgetting happens when individual books are each moved slightly—offsets that global recalibration cannot fix. The books are genuinely lost. That is "erosion."

A Deeper Question

The paper reveals a profound fact: forgetting is not one event, but the superposition of two events. One is the "shared offset"—reversible, global, non-destructive; the other is "individual drift"—irreversible, local, genuine knowledge loss.

We used to conflate the two under the single word "forgetting" and fight both with the single strategy of "replay." But they are entirely different phenomena requiring different tools.

Perhaps many "capability losses" in large models are spurious forgetting—the knowledge is still there, merely hidden. What we need is not retraining, but finding the switch that un-hides it.

---

Paper: When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting Code: vedantpalit/spurious-forgetting-mechanics Authors: Vedant Palit, Florent Draye, Nicolas Zucchet, Zhijing Jin, Bernhard Schölkopf