Machine Unlearning Isn't Just Parameter Rewinding — Optimizer States Must Align with Counterfactuals

Forum topic · 小凯 · 2026-05-19

Summary

A theoretical paper by Stewart (arXiv:2605.17590) argues that machine unlearning has been treated too narrowly as a parameter-correction problem. For optimizers that carry internal memory — specifically online L-BFGS, which maintains curvature pairs approximating the inverse Hessian — deleting training samples changes not only the final weights but also the optimizer's compressed representation of the data stream. Training onward from corrected parameters but stale optimizer state can therefore diverge from the counterfactual model trained without the deleted data. The paper introduces state-aware unlearning metrics: parameter error, memory-operator error (Hessian approximation error), combined state error, and update-direction error, and derives recursive upper bounds on counterfactual state deviation under convexity assumptions. The forum post highlights the core conclusion — unlearning is an optimizer-state alignment problem, not merely a parameter problem — while raising open questions: deep learning losses are non-convex, and online L-BFGS is rarely used in practice compared to Adam, so whether the theory extends to Adam's adaptive momentum state remains unclear.

What the paper is about

Machine unlearning asks: given a trained model, can we produce a model that behaves *as if* certain training samples were never seen — without full retraining?

Existing work largely treats this as parameter correction: adjust the weights so the model's outputs on forgotten data match those of a model that never saw it. Stewart's theoretical work (arXiv:2605.17590) argues this framing is incomplete for optimizers with internal memory.

The core argument

Online L-BFGS maintains a memory of recent curvature pairs used to approximate the inverse Hessian. This memory is a compressed representation of the historical data stream. When you delete training samples:

the final parameters would differ, and
the optimizer's memory would differ in the counterfactual run.

If you only correct parameters and leave the optimizer state untouched, any subsequent continued training starts from a *misaligned state* relative to the counterfactual.

Contributions

The paper proposes state-aware unlearning metrics:

1. Parameter error 2. Memory-operator error (deviation of the Hessian approximation) 3. Combined state error 4. Update-direction error

Under convexity assumptions, it derives recursive upper bounds on counterfactual state deviation.

Key takeaway: unlearning is not a parameter-correction problem — it is an optimizer-state alignment problem.

Open questions (from the forum discussion)

The analysis assumes convexity, while deep-learning losses are non-convex — do the bounds still apply?
Online L-BFGS is not widely used in deep learning (Adam dominates). Can the theory extend to Adam's adaptive momentum state, which also encodes data history?

References

1. Stewart, K. (2026). *Form and Function: Machine Unlearning as a Problem of Misaligned States*. arXiv:2605.17590 [cs.LG]. 2. Bourtoule, L., et al. (2021). *Machine Unlearning*. IEEE S&P. 3. Liu, D. C., & Nocedal, J. (1989). *On the Limited Memory BFGS Method for Large Scale Optimization*. Mathematical Programming.