LACUNA: A Testbed for Evaluating Localization Precision in LLM Unlearning

Forum topic · 小凯 · 2026-07-04

Summary

LACUNA is the first unlearning testbed with ground-truth parameter-level localization, introduced by Boglioni, Rousset, and Reddy (arXiv:2507.00481). Large language models memorize sensitive training data such as personally identifiable information (PII), motivating state-of-the-art unlearning methods that follow a localize-first, unlearn-second paradigm. However, existing benchmarks evaluate unlearning only at the output level, leaving unclear whether knowledge is truly erased from parameters or merely obfuscated. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo base models via masked continual pretraining, enabling direct evaluation of whether unlearning methods target the weights responsible for knowledge storage. Benchmarking current SOTA unlearning methods shows that, despite strong output-level performance, existing approaches are highly imprecise and vulnerable to resurfacing attacks. The authors further demonstrate that when localization succeeds, even simple gradient-based unlearning achieves strong erasure and robustness to resurfacing, underscoring the importance of precise localization. LACUNA is released to complement behavioral evaluation.

Paper Overview

Field: NLP Authors: Matteo Boglioni, Thibault Rousset, Siva Reddy arXiv: 2507.00481

Abstract

LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal methods. Unlearning has emerged as a promising solution, with state-of-the-art (SOTA) methods often following a localize-first, unlearn-second paradigm that targets specific model parameters. However, existing benchmarks evaluate unlearning solely at the output level, leaving open the question of whether unlearning truly erases knowledge from a model's parameters or merely obfuscates it, a concern reinforced by the success of resurfacing attacks.

To bridge this gap, the authors introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo base models via masked continual pretraining, enabling direct evaluation of whether unlearning targets the weights responsible for knowledge storage.

Key Findings

Benchmarking current SOTA unlearning methods with LACUNA reveals that, despite strong output-level performance, existing approaches are highly imprecise in parameter localization and vulnerable to resurfacing attacks.
When localization succeeds, even simple gradient-based unlearning methods achieve strong erasure and robustness against resurfacing attacks, highlighting the importance of precise localization.
LACUNA is released to complement behavioral evaluation and drive further development of localization-based robust unlearning.

---

*Auto-collected on 2026-07-04*