Not All Forgetting Requires a Price: The Surgical Blade of LLM Unlearning

A Surgical Metaphor

Imagine you have surgery to remove a tumor. Afterward you find your sense of taste has changed, you walk with a limp, and your memory has declined. The doctor says: "These are necessary costs of the operation."

But is that really true? Could some of these side effects be "redundant damage" — healthy tissue the scalpel cut unnecessarily? If the excess could be "restored," the tumor would still be removed while your taste and gait recover.

This is not a medical story. It is what an October 2026 paper, "Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning" (arXiv: 2610.11915), is doing — except the patient on the operating table is a large language model.

What Is "Unlearning"

Large language models absorb enormous knowledge during pretraining, some of it harmful — details of bioweapons, private personal data, copyrighted content. "Unlearning" means making the model "forget" this designated knowledge while preserving all other capabilities.

The task is much harder than it sounds. The difficulty is not forgetting — gradient descent can easily raise the loss on a dataset. The difficulty is forgetting *only* what should be forgotten. In practice, unlearning methods cause collateral damage: while forgetting bioweapon knowledge, a model may also forget basic chemistry or how to speak coherently.

Existing methods mitigate this with retain objectives, scope restrictions, or knowledge editing. But the paper's core finding is that even with these constraints, unlearning introduces substantial "recoverable drift" — damage that is not necessary and can be reversed.

Core Finding: Not All Damage Is Necessary

The authors apply existing unlearning methods (NPO, RMU, POO) and measure:

1. Forgetting degree: whether target knowledge is forgotten 2. Internal displacement: how far internal representations have drifted

They then attempt to "recover" using a method called PTP-U, pulling internal representations back. The results:

• Forgetting degree barely changes — target knowledge remains forgotten
• Non-target capabilities recover significantly — general abilities rebound

This means a substantial portion of the damage in existing methods is redundant perturbation — introduced by the optimization process but not required for forgetting. Restoring it preserves forgetting while returning the model close to its original state.

In surgical terms: the tumor is gone, but the scalpel also cut healthy tissue. Sew the tissue back, and the tumor does not regrow.

PTP-U: Propose, Then Project

The proposed method, PTP-U (Propose-Then-Project Unlearning), has two steps:

Step 1: Propose. A local analytic edit "proposes" a modification direction in the model's hidden space, using local geometry of the target knowledge to move the model away from it. Instead of global optimization like gradient descent (which moves everything), it uses a first-order Taylor expansion to approximate local behavior — like a scalpel cutting only a local region.

Step 2: Project. The proposed modification is projected onto a "retain subspace" built from non-target knowledge — the directions that should not be modified. The projection is orthogonal: the modification vector is decomposed into a parallel component (inside the retain subspace) and an orthogonal component. Only the orthogonal component is kept — only modifications that do not conflict with retained capabilities are applied.

This has an elegant mathematical property: if the forgetting direction and retained capabilities are orthogonal, the projection changes nothing — forgetting is unaffected. If they overlap, the projection removes the overlap — retained capabilities are restored. The projection strength adapts to the degree of conflict.

Experimental Results

Tested on four benchmarks:

Forgetting (higher is better):

• WMDP (biosecurity/cybersecurity): 81.22% - 91.03% forgetting rate
• MUSE-Books: lowest BLEU and ROUGE-L scores (most thorough forgetting)

Retention (higher is better):

• MMLU general capability: 94.20% average retention
• At equal forgetting levels, PTP-U's non-target retention consistently exceeds all baselines

Recovery:

• Internal displacement drops significantly after recovery
• Forgetting degree essentially unchanged

The key comparison: at the same forgetting level, PTP-U's non-target retention is 3-7 percentage points higher than the runner-up — a significant gap in the unlearning field.

Why This Matters

PTP-U's core insight is not "a better optimization algorithm" but a redefinition of the problem.

Prior methods ask: "How do we forget target knowledge without harming other capabilities?" This assumes forgetting and retention are two objectives in one optimization, requiring trade-offs.

PTP-U asks instead: "Which modifications are necessary for forgetting, and which are redundant perturbation?" This decomposes the change into necessary and redundant parts — keep the necessary, discard the redundant.

This shares a philosophy with decoupled judge-gate designs: don't optimize two objectives at one level; decouple them across levels. PTP-U's "propose" is the judgment; the "project" is the gate.

A Deeper Analogy

PTP-U suggests a deeper analogy: the relationship between unlearning and model editing.

Model editing precisely modifies one knowledge point; unlearning deletes a class of knowledge. These were seen as different tasks. But PTP-U's mathematical form reveals a unified view: both are "constrained optimization in hidden space." Editing modifies a point while constraining other points; unlearning modifies a direction while constraining other directions. The propose-then-project framework applies to both.

The authors note this connection: "PTP-U connects model editing and teacher-guided distillation through a common constrained objective." This unified view may foreshadow a more general "model modification" framework — covering unlearning, editing, continual learning, alignment fine-tuning, and more.

Limitations

The paper acknowledges several limitations:

1. Local curvature approximation has limited accuracy; first-order Taylor expansion may fail for large modifications. 2. Constraints are defined on sampled knowledge units, not guaranteed to minimize behavioral drift globally; untested retrieval paths may exist. 3. Multimodal settings are harder — cross-modal knowledge sharing makes separating target from non-target more difficult.

These point toward future work: higher-order nonlinear approximations, semantic (not just geometric) constraints, and multimodal extensions.

Conclusion

"Not Every Change Is Necessary" — the title itself is a reminder to the whole field. We previously assumed forgetting necessarily carries a cost and worked to minimize it. This paper says: first ask which costs are necessary, then remove the unnecessary ones.

The deeper implication: modifications introduced by the optimization algorithm are not the same as modifications required by the task. Gradient descent is a blunt tool — it changes any parameter that lowers loss, whether or not the task requires it. PTP-U's projection step is essentially a filter: keep task-required modifications, filter out redundant perturbation.

This insight may extend broadly. Any "model modification" task — alignment, fine-tuning, continual learning — may contain redundant perturbation. Propose first, then project, keeping only necessary changes — this paradigm may be more fundamental than "adding a regularization term."

Tissue cut unnecessarily by the scalpel can be sewn back. Parameters changed unnecessarily by gradient descent can be projected back.

---

Paper: https://arxiv.org/abs/2610.11915 HTML version: https://arxiv.org/html/2610.11915v1