Notes from an Auditor
Suppose you are a GDPR auditor. A company claims to have "forgotten" user A's data and submits an evaluation report: forgetting-set accuracy d_f = 0.02, retention-set accuracy d_r = 0.95, gap |d_f - d_t| = 0.0414, below the 0.05 threshold—pass.
You sign off.
But wait. The model you're holding comes with more than weights: a set of batch normalization running mean / running var statistics. These were accumulated online during training, no one recorded which data they were fit on, and no one checked whether they still match the current weights. They just sit quietly in the checkpoint file, like a ledger nobody audits.
Junlong Shen and Xingyu Li (University of Alberta), in arXiv 2609.11490, did something simple but unprecedented: they took the BN statistics from 263 publicly released unlearning checkpoints and refit them on the retention set with weights completely untouched (bit-identical), then checked whether the published numbers move.
Result: of 221 checkpoints, 47 showed forgetting-accuracy drift beyond the ±1.2pp boundary. Of these, 12 pass/fail verdicts flipped outright—8 originally passing became failing, 4 originally failing became passing.
This is not a bug; it's a feature. Or rather, a variable nobody in the field has recorded.
The Core Question: The Numbers Moved—But Why?
The basic logic of unlearning evaluation: train a model, remove certain data and retrain (unlearned), then compare against a reference model trained from scratch on the retention set only (retrained reference). If the two perform similarly on the forgetting set, unlearning is deemed successful.
But there's a hidden variable. Both models carry BN statistics accumulated online during training. The unlearned model's statistics were accumulated during the unlearning process; the retrained reference's were accumulated from scratch on the retention set. The two sets of statistics were fit on different data, but nobody records this difference.
The paper runs a key experiment to distinguish two explanations:
Explanation A (the intuitive one): the BN statistics retain residual information about deleted data, so refitting changes the numbers.
Explanation B (the actual one): the BN statistics didn't keep up with weight changes during unlearning—they became "stale." Refitting merely "calibrates" them back to the state matching the current weights.
The experimental design is precise: fix the weights, the estimator, the image transforms, and the fitting pool size; vary only the source of k records in the fitting pool—one group from the retention set, one from the deleted set.
Results:
| Fitting pool source | Forgetting accuracy drift (pp) | |---|---| | Deployment share of deleted data | +0.015 | | Entirely deleted data | +0.057 | | 1.2pp boundary | 1.2 |
The "ghost" of deleted data is nearly absent. Even stuffing all deleted records into the fitting pool yields only 0.057pp of drift, far below the 1.2pp boundary. No checkpoint exceeded 0.5pp.
But another measurement was significant: how far each checkpoint's shipped BN statistics sit from any "retention-set refit" state. All 20 checkpoints tested passed the test of being beyond the null-hypothesis band, with a median distance 5.43× the provenance distance.
In other words: the numbers moved not because the deleted data still lives on, but because the BN statistics failed to track the weight changes and went stale.
"Stale, Not Surviving": A Crucial Distinction
This is the paper's sharpest insight.
The field's implicit assumption is: if numbers still move after unlearning, the deleted data "survives" in the model. This paper says: no—the numbers may move simply because a statistic went stale, with no relation to data survival.
It's like changing a lock but leaving the old key's imprint on the door frame. You think someone can still open the door with the old key (data surviving), but really the frame just wasn't replaced (stale statistics). Replacing the frame (refitting BN) doesn't require changing the lock (retraining weights), yet the frame's state does affect whether "the door closes tightly" (evaluation numbers).
The paper offers a more precise framing: BN statistics are a "deployed state," not a "training artifact." They are runtime configuration needed at deployment, separate from the weights. But the entire field treats them as part of the weights and never audits them separately.
"Per-Checkpoint, Not Per-Method": The Failure of Method-Level Verdicts
Another key finding: drift is a checkpoint-level property, not a method-level property.
One method family's average drift was +0.830pp, below the 1.2pp boundary—so the method itself looks "safe." But 8/45 checkpoints in that family exceeded 1.2pp.
This is Statistics 101: averages don't describe individuals. Yet all unlearning evaluation is method-level: papers report method A's average forgetting accuracy on dataset D, and readers judge method A accordingly.
The paper's point: an auditor holds a checkpoint, not a method. A method-level verdict ("method A's average drift is below the boundary") cannot tell you whether the specific checkpoint in your hands is reliable. It's like saying "this batch of pills meets average potency standards"—that doesn't guarantee the pill in your hand does.
12 Flipped Verdicts: Concrete Examples
The paper lists every verdict flip. A few stand out:
Example 1: MU-Bench bad-teaching, 2% removal
- Published: |d_f - d_t| = 0.0414 (pass, < 0.05)
- After refit: 0.0573 (fail, > 0.05)
- Verdict: "unlearning succeeded" flips to "unlearning failed"
- Published: |d_f - d_t| = 0.0503 (fail)
- After refit: 0.0437 (pass)
- Verdict: "unlearning failed" flips to "unlearning succeeded"
- Published: RA = 0.9497 (fail, < 0.95)
- After refit: 0.9856 (pass)
- Verdict: "unlearning failed" flips to "unlearning succeeded"
- Weights (trainable tensors)
- BN running statistics (deployed state)
- Optimizer state (if present)
- Quantization scales (if quantized)
- Other metadata
Example 2: MU-Bench random-label, 8% removal
Example 3: Unlearning Comparator, SalUn, class automobile
Note: flips occur in both directions. This is not systematic bias—it's noise. Stale BN statistics can make a model look like it "didn't forget enough" (numbers too high) or "forgot too much" (numbers too low).
State of the Field: 52 Papers, Only 4 Mention This Variable
Appendix K of the paper contains a literature audit: 52 unlearning papers, 42 passing a filter (mentioning "unlearn" at least 15 times), of which 19 use convolutional backbones with BN and compare against a retrained reference.
Of those 19, only 4 mention deployed normalization state. Three mention it in passing without treating it as a comparison variable; one is concurrent work (Jawandhia et al., 2026) dedicated to this channel.
In other words: 95% of relevant papers in the field are comparing on an uncontrolled variable.
Boundaries: Where the Finding Stops
The paper is honest about its limits:
1. Applies only to BN backbones. LayerNorm and RMSNorm (including all LLMs) carry no running statistics, so this channel doesn't exist for them. The paper tested a second "deployed state"—quantization scales—on 118 quantized checkpoints (including 47 swin-base transformers) and found zero drift. The effect is BN-specific, not general to all deployed states.
2. It's an existence result, not a rate. 12 flipped verdicts among 263 checkpoints does not mean "unlearning evaluation has a 4.6% error rate." The paper explicitly frames this as an existence proof: at the field's own operating points, on published artifacts, verdicts do flip.
3. The census is a lower bound, not an estimate. Because the statistical method is conservative (requiring all 10 replicates to agree in direction), the actual number of drifting checkpoints may be higher.
Engineering Takeaways: What Should an Auditor Do?
The paper's conclusions are practical:
1. Record the fitting convention at release. A checkpoint release should state which data, transforms, and accumulation rules produced its BN statistics. Nobody records this today.
2. Audit at the checkpoint level, not the method level. Method-level verdicts cannot describe the artifact in your hands.
3. Refitting is a cheap diagnostic. Fixing the weights and refitting BN statistics on the retention set takes only minutes. Large post-refit drift indicates stale BN statistics.
4. Don't extrapolate to LLMs. LLMs use LayerNorm/RMSNorm with no running statistics, so this channel doesn't exist for them. But whether LLMs' other "deployed states" (e.g., accumulated KV cache statistics) suffer similar issues is an open question.
Personal Reflection: The Convergence of Three Conceptual Lineages
This paper lands precisely at the intersection of three lineages I've long tracked:
"Evaluation blind-spot law" gains another instance: 95% of relevant papers compare on an uncontrolled variable (the BN fitting convention). This isn't one paper's oversight; it's a systemic blind spot—like omission blindness (LLM judges can't detect missing content) and Moral Geometry (models learn corpora, not theory): evaluation doesn't measure what you think it measures.
"Scalar illusion" gains another instance: the method-level verdict is a scalar (average drift +0.830pp), but the actual property is a per-checkpoint vector (8/45 over the boundary). Managing a vector with a scalar is like measuring blood pressure with a thermometer—the same distortion as compressing high-dimensional properties into one dimension.
"Judgment-gate decoupling" gains another instance: the checkpoint's weights are correct (judgment passes), but the BN statistics are stale (the gate didn't close). Published numbers look fine while the underlying state is inconsistent—like LLM judges that recognize but cannot reverse, or PoP's middle layers knowing the answer is wrong while the output layer never consults them: judgment failing to propagate to execution.
New concept: "Stale, not surviving." The paper's most original insight: moving numbers do not mean the data is alive. The field assumed "numbers move = data survives," but it may just be "numbers move = statistics went stale." This distinction directly affects the credibility of the unlearning evaluation literature.
New concept: "Deployed state vs. training artifact." BN statistics are deployed state, not a training artifact—runtime configuration separate from the weights. The entire field treats them as part of the weights and never audits them separately. This distinction may extend to other things "attached to checkpoints but not weights": quantization scales, KV cache statistics, adapter initialization states, and more.
A Deeper Analogy
The paper suggests an even deeper analogy: a checkpoint is a complete deployment package, not just weights.
What's in a checkpoint file?
It's like buying a computer: the CPU is fine (weights), but the BIOS settings are factory defaults (BN statistics). The machine boots, but performance may be off. You can't say "the CPU is fine, so the computer is fine," because BIOS settings are part of the deployment.
The unlearning field has been auditing CPUs; nobody audits the BIOS. This paper says: BIOS settings go stale, and stale enough to drift evaluation numbers until verdicts flip.
Conclusion
This paper did something simple and unprecedented: audit the BN statistics of 263 publicly released unlearning checkpoints. It found 47 whose numbers drift, 12 whose verdicts flip—due to stale statistics, not surviving data. 95% of relevant papers in the field have never recorded this variable.
This is not a "unlearning doesn't work" conclusion. The paper is clear: most checkpoints are stable, most method-level verdicts are reliable. But "most" is not "all," and an auditor holds a specific checkpoint, not a method average.
The paper's final line is well put: "A release should therefore name the fitting convention beside the number, on the batch-normalized backbones that ship one."
Write the convention next to the number. It's the minimum requirement—and nobody is doing it yet.
---
Paper: arXiv 2609.11490 — Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints Authors: Junlong Shen & Xingyu Li (University of Alberta) Published: 2026-09-10