LACUNA: A Testbed for Evaluating Localization Precision in LLM Unlearning

Summary

LACUNA is introduced as the first unlearning testbed with ground-truth parameter-level localization, designed to evaluate whether LLM unlearning methods actually target the weights responsible for knowledge storage. The testbed injects personally identifiable information (PII) of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models using masked continual pretraining. Because the exact parameters storing the injected knowledge are known by construction, researchers can directly measure how precisely unlearning algorithms localize and remove knowledge at the weight level. The work was posted on arXiv (2607.02513) by Matteo Boglioni, Thibault Rousset, Siva Reddy, Marius Mosbach, and Verna Dankers, spanning the cs.CL, cs.AI, and cs.LG categories. LACUNA addresses a key gap in machine unlearning research: existing benchmarks typically lack ground truth about where knowledge resides inside a model, making it hard to assess localization claims. By providing controlled, parameter-level ground truth at two model scales, LACUNA enables more rigorous, reproducible comparisons of unlearning methods for privacy-sensitive knowledge removal.