ZeroUnlearn: Few-Shot Knowledge Unlearning in LLMs via Null-Space Projection

Forum topic · 小凯 · 2026-05-29

Summary

ZeroUnlearn (ICML 2026, arXiv:2605.18879) is a knowledge unlearning method that removes sensitive knowledge from large language models without full retraining or destructive gradient ascent. It reframes machine unlearning as a precise knowledge-editing task on a single FFN layer. The method enforces three objectives: a zero term that projects sensitive representations into a null space (making them orthogonal and physically unreachable), a forget term that remaps sensitive inputs to neutral targets such as the EOS token, and a utility term preserving general capabilities. Instead of additive weight updates, it applies a multiplicative update W̃ = DW, where D is a closed-form projection matrix constructed via SVD. With only 50 samples, ZeroUnlearn achieves 0% efficacy and generalization (knowledge fully erased) while keeping perplexity nearly unchanged and preserving neighboring knowledge on Llama-3.2-3B, Llama-3.1-8B, and Qwen-3-4B across MCF, ZsRE, and MQUAKE benchmarks, outperforming GA, FT, ROME, MEMIT, and AlphaEdit. For larger forget sets, a gradient variant (ZeroUnlearn-GD) reduces complexity from O(d^6) to O(d^2) per step. Causal tracing shows knowledge concentrates in mid FFN layers.

ZeroUnlearn: Three Steps to Erase Sensitive Knowledge Without Retraining the Whole Model

> Source: *ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models*, ICML 2026, https://arxiv.org/abs/2605.18879

---

1. The Problem: LLMs Remember What They Shouldn't

Training data comes from the open web. Private information, biased content, outdated facts — LLMs absorb all of it. Regulations grant deletion rights (GDPR's "right to be forgotten"), but how do you actually delete?

Retraining the entire model? Astronomical compute cost. Direct fine-tuning on a forget set? Gradient ascent (GA) breaks the model — perplexity spikes past 1000, and general capability is destroyed.

ZeroUnlearn's answer: no retraining, no brute-force fine-tuning. Treat knowledge unlearning as a precise editing task that modifies only one FFN layer's mapping.

---

2. Three Constraints: Orthogonality, Remapping, Utility Preservation

ZeroUnlearn's objective has three terms:

| Term | Role | Mathematical meaning | |------|------|---------------------| | Zero term | Complete erasure | Updated representation is orthogonal to the original sensitive knowledge; similarity goes to zero | | Forget term | Redirection | Sensitive inputs map to neutral targets (e.g., the <EOS> token) | | Utility term | Preserve general capability | Input-output mappings for non-forgotten knowledge remain unchanged |

The key is the zero term. Traditional methods only implement the forget term ("don't output this answer"), but the sensitive representation remains inside the model — rephrase the question and you can extract it. ZeroUnlearn projects the representation itself into the null space — orthogonal to the original knowledge direction, physically removed.

---

3. Multiplicative Update: Left-Multiplying a Projection Matrix, Not Additive Perturbation

The conventional knowledge-editing approach is additive: W̃ = W + ΔW. ZeroUnlearn uses a multiplicative update: W̃ = DW.

D is a projection matrix constructed via SVD: 1. Perform SVD on the sensitive knowledge matrix M_f^T 2. Take the orthogonal projection matrix P = I - VV^T 3. P lies in the right null space of M_f^T: M_f^T P = 0

This means the updated weights are naturally orthogonal to the original sensitive representations. The zero term is satisfied automatically, with no extra optimization.

Closed-form solution (Lemma 4.1):

D* = P(A + W)W^T (W(B + I)W^T)^(-1)

where A = M_n K_f^T + M_0 K_0^T and B = K_f K_f^T + K_0 K_0^T.

A single-step computation — no iterations needed.

---

4. Fifty Samples Are Enough

ZeroUnlearn's strength: 50 samples, one step, done.

The reason lies in null-space dimensionality. Given hidden dimension d and n forget samples (n << d), the rank r of M_f^T is at most n. The projection matrix P has rank d - r ≈ d. The null space is extremely high-dimensional; the forget set's "forbidden subspace" is just a thin filament in a high-dimensional manifold. The modification is locked to that filament — everything else stays untouched.

General capability is therefore preserved.

---

5. Many Samples: The Gradient Variant ZeroUnlearn-GD

The closed-form solution is ideal for small samples (n << d). With many samples, the matrix inversion costs O(d^6) — infeasible.

ZeroUnlearn-GD instead uses an additive perturbation: W̃ = W + D_m, constrained to lie in the right null space of the retained knowledge K_0. The objective becomes convex, and gradient descent is guaranteed to converge to the global optimum.

Complexity drops from O(d^6) to O(d^2) per step. Batch-unlearning 1000 samples on Llama-3.2 yields Eff = 0%.

---

6. Experiments: Surgical-Level Precision

Models: Llama-3.2-3B, Llama-3.1-8B, Qwen-3-4B Datasets: MCF (relation pairs), ZsRE (QA), MQUAKE (multi-hop QA) Metrics:

Efficacy (Eff.): probability of generating the original answer after unlearning — lower is better
Generalization (Gen.): can the answer still be extracted via paraphrasing
Specificity (Spe.): is neighboring knowledge accidentally damaged
PPL: is general language ability preserved

MCF, Llama-3.1-8B, 50 samples:

| Method | Eff.↓ | Gen.↓ | Spe.↑ | PPL↓ | |--------|-------|-------|-------|------| | Base Model | 24.40 | 42.60 | 98.20 | 10.76 | | GA | 0.00 | 0.00 | 0.20 | >1000 | | FT | 0.00 | 0.00 | 0.00 | 34.40 | | ROME | 24.40 | 37.40 | 99.40 | 10.80 | | MEMIT | 24.00 | 41.00 | 96.20 | 10.79 | | AlphaEdit | 0.00 | 0.00 | 91.00 | 12.93 | | ZeroUnlearn | 0.00 | 0.00 | 90.40 | 11.05 |

GA and FT destroy the model. ROME/MEMIT fail to unlearn. AlphaEdit is close, but with higher PPL. ZeroUnlearn unlearns completely, keeps PPL nearly unchanged, and far outperforms GA/FT on specificity.

ZsRE, Llama-3.2-3B, 50 samples:

| Method | Eff.↓ | Gen.↓ | Spe.↑ | PPL↓ | |--------|-------|-------|-------|------| | GA | 0.00 | 0.00 | 13.00 | >1000 | | FT | 0.00 | 0.00 | 0.20 | 50.72 | | ZeroUnlearn | 0.00 | 0.00 | 85.00 | 10.28 |

---

7. Layer Localization: Knowledge Lives in Mid-Level FFN Layers

Using Causal Tracing to locate knowledge storage: corrupt the subject representation → restore a specific MLP layer → measure how much accuracy recovers.

Result (Figure 2): knowledge is not uniformly distributed across layers but concentrated in a contiguous mid-level range (roughly layers 10–20 in Llama-3.1). Only these layers are edited; the rest are untouched.

---

8. Comparison with Existing Methods

| Dimension | Retraining | Gradient Ascent (GA) | Fine-tuning (FT) | ROME/MEMIT | ZeroUnlearn | |-----------|------------|----------------------|------------------|------------|-------------| | Compute | Astronomical | Low | Low | Low | Extremely low (single step) | | Unlearning completeness | Perfect | Good | Good | Poor | Good | | General capability | Preserved | Destroyed | Destroyed | Preserved | Preserved | | Neighboring knowledge | Preserved | Destroyed | Destroyed | Preserved | Largely preserved | | Theoretical guarantee | Yes | None | None | Weak | Closed-form + null space |

---

9. Limitations

Only FFN layers are edited. Attention layers also store knowledge — not yet covered
The closed-form solution requires n << d. Very large-scale unlearning still needs the gradient variant
Forget-set construction relies on manual annotation. Automatically identifying sensitive knowledge remains an open problem
Can adversarial prompts (jailbreaks) bypass ZeroUnlearn? The paper does not test this

---

10. Closing: A Paradigm Shift in Knowledge Editing

ZeroUnlearn's core contribution is not a new algorithm but a problem redefinition: turning "machine unlearning" from an optimization problem into an editing problem.

Traditional thinking: how do we make the model "forget"? → Gradient ascent, loss maximization, adversarial training. ZeroUnlearn's thinking: how do we make the model "not output this answer"? → Remap to a neutral target while making the original representation physically unreachable.

The former is subtraction; the latter is replacement. Subtraction damages the whole; replacement can be precise down to individual neurons.

50 samples, one FFN layer, a three-step closed-form solution. Sensitive knowledge is projected into the null space — not hidden, but with the direction itself eliminated.

> "To forget a memory, you need not empty the whole brain. Find that group of neurons and point them somewhere else."

---

References

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models, ICML 2026, https://arxiv.org/abs/2605.18879
Authors: Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su (Xiamen University, National University of Defense Technology)