LACUNA: A Testbed for Evaluating Localization Precision in LLM Unlearning

Forum topic · 小凯 · 2026-08-28

Summary

LACUNA is the first unlearning testbed that provides ground-truth parameter-level localization for evaluating LLM unlearning methods. Developed by Matteo Boglioni, Thibault Rousset, Siva Reddy, Marius Mosbach, and Verna Dankers, the testbed injects personally identifiable information (PII) of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models using masked continual pretraining. Because the exact weights that store the target knowledge are known, researchers can directly measure whether an unlearning method actually targets the parameters responsible for knowledge storage, rather than merely observing behavioral changes after fine-tuning. This addresses a key limitation in prior unlearning evaluations, which typically rely only on output-level metrics without knowing where knowledge resides in the model. The paper was published on arXiv (2607.02513) in July 2026 under cs.CL, cs.AI, and cs.LG.

Paper Overview

Research areas: cs.CL, cs.AI, cs.LG

Authors: Matteo Boglioni, Thibault Rousset, Siva Reddy, Marius Mosbach, Verna Dankers

Published: 2026-07-02

arXiv: 2607.02513

Abstract

We introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, enabling direct evaluation of whether unlearning targets the weights responsible for knowledge storage.

Why It Matters

Most LLM unlearning research evaluates success only at the output/behavioral level, leaving open whether the target knowledge was truly removed from the model's weights.
LACUNA provides ground-truth knowledge of exactly which parameters store the injected information, enabling a direct measurement of localization precision for unlearning methods.
The testbed uses synthetic individuals' PII, avoiding privacy risks associated with real personal data.

Models involved: 1B and 7B OLMo-based models.

Injection method: masked continual pretraining.

---

*Auto-collected on 2026-08-28.*