47 of 263 Checkpoints Show Digital Drift: The Unrecorded Variable in Unlearning Evaluation
This post is a technical commentary on arXiv paper 2609.11490 — *Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints* — by Junlong Shen and Xingyu Li (University of Alberta, published 2026-09-10).
An Auditor's Scenario
Imagine you are a GDPR auditor. A company claims it has "unlearned" user A's data and submits an evaluation: forgetting-set accuracy gap |d_f − d_t| = 0.0414, below the 0.05 threshold — pass. You sign off.
But the checkpoint also ships batch normalization running mean/var statistics, accumulated online during training, with no record of what data they were fit on and no check that they match the current weights. They sit in the checkpoint file like an unexamined ledger.
The Core Experiment
The authors took 263 publicly released unlearning checkpoints, kept the weights bit-identical, re-fitted the BN statistics on retain-set data, and observed whether published numbers changed.
Result: 47 of 221 checkpoints drifted beyond a ±1.2pp boundary on forgetting accuracy. 12 pass/fail verdicts flipped — 8 from pass to fail, 4 from fail to pass.
The paper distinguishes two explanations:
- Explanation A (intuition): the BN statistics retain traces of the deleted data, so re-fitting changes numbers.
- Explanation B (actual): the BN statistics failed to track weight changes during unlearning and became *stale*; re-fitting merely recalibrates them to the current weights.
- MU-Bench bad-teaching, 2% removal: published |d_f − d_t| = 0.0414 (pass) → refit 0.0573 (fail)
- MU-Bench random-label, 8% removal: 0.0503 (fail) → 0.0437 (pass)
- Unlearning Comparator, SalUn, class automobile: RA 0.9497 (fail) → 0.9856 (pass)
By varying only the source of k records in the fitting pool while fixing weights, estimator, transforms, and pool size:
| Fitting pool source | Forgetting accuracy drift (pp) | |---|---| | Deployed share of deleted data | +0.015 | | All deleted data | +0.057 | | Boundary | 1.2 |
The "ghost" of deleted data is nearly absent — no checkpoint exceeded 0.5pp. Meanwhile, a separate measurement showed all 20 tested checkpoints carry BN statistics far from any retain-set-refit state, with median distance 5.43× the provenance distance.
Key Insight: "Stale, not surviving"
The field's implicit assumption: if numbers move after unlearning, the removed data "survives" in the model. The paper says: no — numbers may move simply because a statistic went stale. BN statistics are a deployed state, not a training artifact: a runtime configuration separate from the weights, yet treated as part of the model and never audited separately.
"Per-checkpoint, not per-method"
Drift is a checkpoint-level property, not a method-level one. One method family's mean drift was +0.830pp (below the 1.2pp boundary), yet 8 of its 45 checkpoints exceeded the boundary. Method-level verdicts cannot tell an auditor whether the specific artifact in hand is reliable — like saying a batch of pills meets dosage standards on average.
Verdict Flips Cut Both Ways
Field Status
A literature audit of 52 unlearning papers (19 using BN backbones with retrained references) found only 4 mention deployed normalization state — and 3 of those only in passing. 95% of relevant papers compare on an uncontrolled variable.
Boundaries of the Finding
1. BN-only: LayerNorm and RMSNorm (all LLMs) carry no running statistics. A second deployed state — quantization scales — showed zero drift across 118 quantized checkpoints (including 47 swin-base transformers). The effect is BN-specific. 2. Existence, not rate: 12 flips among 263 checkpoints is an existence proof, not an error-rate estimate. 3. Census is a lower bound: the conservative statistical method (requiring 10 same-direction replicates) may undercount drift.
Engineering Recommendations
1. Record the fitting convention at release — what data, transforms, and accumulation rules produced the BN statistics. 2. Audit at checkpoint level, not method level. 3. Re-fitting is a cheap diagnostic — a few minutes with fixed weights on retain data; large drift indicates stale statistics. 4. Do not extrapolate to LLMs — though other deployed states (e.g., KV cache statistics) remain an open question.
Commentary in the Post
The author situates the paper at the intersection of three conceptual lineages they track: "evaluation blind-spot" (95% of the field comparing on an uncontrolled variable), "scalar illusion" (method-level averages flattening per-checkpoint vector outcomes), and "judgment-gate decoupling" (correct weights, stale runtime state). The post coins/adopts two concepts: "stale, not surviving" (numbers moving ≠ data alive) and "deployed state vs. training artifact" (checkpoints ship a full deployment package — weights, BN statistics, quantization scales, metadata — not just weights). The analogy: the field audits the CPU but never the BIOS, and a stale BIOS can swing evaluation numbers past verdict thresholds.
Conclusion
Most checkpoints are stable and most method-level verdicts hold. But "most" is not "all," and an auditor holds a specific checkpoint, not a method average. The paper's closing line: "A release should therefore name the fitting convention beside the number, on the batch-normalized backbones that ship one."
---
Paper: arXiv 2609.11490 — Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints Authors: Junlong Shen & Xingyu Li (University of Alberta) Published: 2026-09-10