The Model "Forgot" Harry Potter? Plot Its Activation Vectors and the Lie Falls Apart
A Problem That Keeps AI Safety Engineers Up at Night
You trained a large model and it "knows" Harry Potter—that's normal, the pretraining corpus is full of it. But legal says your production model can't output copyrighted character information. So you run machine unlearning, and the model now answers "Who is Harry Potter?" with "I don't know" or irrelevant filler. ROUGE-L is near 0; unlearning succeeded, right?
Then an attacker arrives. Using activation steering—injecting a direction into the residual stream—the model immediately starts chattering about Hogwarts again. Your "forgotten" knowledge was just a sticky note reading "don't mention Harry Potter" slapped over untouched weights.
This is not hypothetical. From 2024–2026, machine unlearning has repeatedly exposed the same gap: a huge distance between behavioral metrics (what the model outputs) and parametric knowledge (what the weights store). ROUGE-L of 0 doesn't mean the knowledge was deleted—only that the model learned to shut up.
How can you tell whether a model truly forgot, or is just pretending?
A paper by Adar Peleg, Dvir Alsheich, Tomer Ashuach, and Avi Mendelson (Technion), published at EMNLP 2026 Findings as *"The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation,"* offers an elegant answer: don't look at what the model says—look at the geometry of its internal activations.
Core Intuition: Knowledge Has a Shape, Ignorance Has Another
A simple truth: if you genuinely know something, questions from different angles all retrieve the same information. Ask "Who is Harry Potter?", "Where is Hogwarts?", or "Who does Voldemort fear?"—all of them activate the same "Harry Potter" conceptual region.
If you've never heard of Harry Potter, these questions only activate unrelated syntactic noise—different directions each time, chaotic.
Knowledge = consistent direction; ignorance = scattered directions.
That is the paper's core hypothesis. Making it quantitative requires filtering out syntactic noise: original activations mix conceptual and grammatical information.
anon-vector: Stripping Syntax via Subtraction
The authors use a trick called anon-vectors (from Seyitoğlu et al. 2024):
1. Take a concept query: q = "Who is Harry Potter?"
2. Build an anonymized variant replacing the entity with a fake name, e.g., a = "Who is Alex Ryden?"
3. The two queries have identical syntax; only the concept entity differs.
4. Pass both through the model, collect intermediate hidden states h(q) and h(a).
5. Subtract: Δv = h(q) - h(a).
The syntax cancels out in the subtraction; what remains is the pure parametric signal for "Harry Potter." Repeat for N paraphrased queries to get N anon-vectors.
- If the model truly knows the concept: the N vectors point in the same direction—a compact, low-entropy cluster.
- If it doesn't: only noise remains, and the N vectors scatter across the high-dimensional space.
- AUSS_L2: mean pairwise Euclidean distance between normalized anon-vectors.
- AUSS_cos²: complement of mean cosine similarity—closer to 0 when directions align.
- AUSS_Jac: Jaccard distance between each vector's top-32 magnitude dimensions.
- Positive class (known): Harry Potter QA pairs—every mainstream LLM saw copious HP text in pretraining.
- Negative class (unknown): fictional authors from the TOFU benchmark (Maini et al. 2024)—created by the benchmark authors, absent from any pretraining corpus.
- GA (Gradient Ascent): negative CE loss on the forget set.
- DPO: forget answers as rejected, retain answers as chosen.
- NPO: negative preference optimization against a frozen reference model.
- NPO+KL: NPO with KL regularization on the retain set.
- RMU: pushes forget-set hidden states toward a random direction at intermediate layers.
- Activation Steering: injecting direction vectors into the residual stream.
- ICL (In-Context Learning): few-shot examples guiding re-emission.
- GCG: gradient-searched adversarial suffixes.
- MIA: membership inference.
- ICL recovery: AUSS_L2 is most predictive (ρ = -0.51). More fragmentation → harder ICL recovery, because ICL needs a coherent direction to latch onto.
- GCG: Reg_S2 and Reg_cos strongest (ρ = +0.45). GCG is gradient search; scattered representations offer more exploitable directions.
- Activation steering: AnonDiff_S2 strongest (ρ = +0.48).
- MIA: no metric significant after BH correction—MIA operates at the distributional level, weakly tied to geometry.
- GCG: peaks at layer 29 (AUSS_Jac, ρ = -0.853)
- ICL: peaks at layer 28 (Batch_Cov_TopEig, ρ = -0.886)
- No token generation—works with APIs that only expose hidden states.
- No pre-unlearning checkpoint—unlike Xu et al. (2025)'s PCA-similarity/CKA, only the post-unlearning model is needed.
- Single forward pass—no GCG (thousands of gradient steps), no steering coefficient search.
- Anonymization is string substitution, sensitive to tokenization (different token lengths for "Harry Potter" vs. "Alex Ryden").
- Requires B ≥ 2 forget-set samples to construct batch directions.
- Concept coverage limited to English fiction/fact domains; non-English and technical domains untested.
- The sign-direction explanation (ICL negative vs. GCG positive) is inferential, not causally validated at the generation level.
- Hardware: 2 × 40GB VRAM GPUs (e.g., 2×L40S or 2×A100-40GB), 64GB RAM, 100GB disk.
- Includes all experiment code, data, and precomputed results.
- Supports
device_map="auto"for large models. - Paper: arXiv:2611.05272 [cs.AI], https://arxiv.org/abs/2611.05272
- Venue: EMNLP 2026 Findings
- Code: https://github.com/AdarPeleg/geometry-of-ignorance
- Authors: Adar Peleg, Dvir Alsheich, Tomer Ashuach, Avi Mendelson (Technion - Israel Institute of Technology)
The authors call this Activation Fragmentation: knowledge is cohesive, ignorance is fragmented.
Quantifying Fragmentation
Three core metrics, collectively AUSS (Activation Uniformity Scatter Score):
Plus AnonDiff_S2, based on the Rényi-2 entropy of the Gram matrix—essentially the "effective dimensionality" the vectors occupy.
No token generation, no attacks—just two forward passes. Can even be reduced to one anonymized variant (K = 1) and a single forward pass.
Experiment 1: Harry Potter vs. Fictional Authors — 10/10 Perfect Separation
To prove the metric measures knowledge itself, the authors used a clean contrastive design:
Result: all 10 instruction-tuned models (Qwen ×4, Gemma ×2, LLaMA ×4, 2B–14B) were perfectly separated on all 9 metrics. The strongest, AUSS_cos², hit Cohen's d = 21.51—an astronomical effect size. A leave-one-model-out logistic regression classifier achieved 100% accuracy (AUC = 1.0).
Three model families, different training recipes, different scales—all consistent.
Experiment 2: Does Knowledge Survive Unlearning?
The critical question: after unlearning, can fragmentation geometry predict whether attackers can recover the knowledge?
They ran 5 unlearning methods × 3 models × 3 concepts = 54 configurations:
The first four mainly act on the output layer—changing what the model says. RMU directly perturbs intermediate representations—changing how the model thinks.
Four attacks tested recoverability:
Key findings:
GA and DPO drove ROUGE-L to near 0 but barely changed fragmentation—nearly identical to the un-unlearned base model. Pure output suppression; the knowledge is intact.
RMU perturbs intermediate representations, so fragmentation shifted somewhat but never fully fragmented—better than GA/DPO, but far from true deletion.
This aligns with Xu et al. (2025) and Schoepf et al. (2026): current unlearning methods at best relocate knowledge, they don't erase it.
Attack Prediction: Which Geometry Predicts Which Attack?
Different attacks exploit different geometric features, at different layers:
Note the reversed signs: ICL is negatively correlated (more fragmentation = harder to attack), while GCG and steering are positively correlated (more fragmentation = easier to attack). The interpretation: ICL needs a coherent conceptual direction to "ride"; GCG and steering search/inject directly in representation space, where dispersion gives attackers more dimensions to work with.
Layer Analysis: Each Attack Has a "Preferred Floor"
One layer apart—both operate near the model's "decision layers." More practically: the best layer is predictable, stably at ~0.8× model depth. A fixed 0.8× rule matches oracle per-model search under leave-one-model-out validation.
Engineering Value: A One-Forward-Pass Unlearning Audit
For AI safety engineers, this is a cheap pre-screening tool:
1. After unlearning, take forget-set queries and build anonymized variants. 2. Two forward passes; grab hidden states at ~0.8× depth. 3. Compute AUSS_L2 and AUSS_cos². 4. If metrics resemble the base model—knowledge remains; skip expensive attacks and declare unlearning failed. 5. If geometry is truly fragmented—run attacks to confirm.
Key advantages:
Limitations:
Open-Source Code
The paper's code is fully open: https://github.com/AdarPeleg/geometry-of-ignorance
My Take: Relation to the Capability–Behavior Gap
This paper connects to a broader lineage—the capability–behavior gap. Recent papers make the same point: model behavior (outputs) is a lower bound on capability, not an equivalence. "Knowing" and "using" can dissociate; "saying" and "believing" can separate.
This paper adds another node: unlearning's behavioral metrics (ROUGE-L) and parametric knowledge (geometry) can fully decouple. GA drives ROUGE-L to 0 while geometry stays untouched—the model "behaviorally forgot" but "parametrically remembers."
The paper goes further than "identifying the problem"—it provides a diagnostic tool. AUSS is the probe (measuring geometry); the four attacks are interventions (testing recoverability). Crucially, AUSS predicts ICL and GCG recovery success—the probe is predictive, not merely descriptive.
The deeper insight: knowledge's presence in a model is not binary (present/absent) but geometric (cohesive/fragmented). A concept's "existence strength" isn't a scalar—it's a shape: directional coherence, effective dimensionality, layer-wise distribution.
For AI safety practitioners: if you want to audit unlearning, don't just read outputs. Plot the activation vectors—if the arrows still point the same way, the knowledge is still there, whatever the model claims.