ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Paper Overview

• Field: NLP
• Authors: Sahil Kale, Ian Harris
• Published: 2026-08-22
• arXiv: 2608.20338

Summary

Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called *unlearning*, yet existing methods and benchmarks fail to evaluate this capability completely.

Current approaches rely on disjoint forget sets and retain sets made up of independent facts, and measure success using simple, direct factual recall. This framing misses a critical requirement of unlearning: the ability to eliminate harmful behaviors while preserving benign, beneficial knowledge.

The paper argues that effective unlearning must operate at the concept level, ensuring complete removal of unsafe applications while maintaining correct and useful uses—achieving conceptually complete and meaningful unlearning.

To this end, the authors introduce the notion of dual-use concepts: concepts that can be used both harmfully and benignly. Based on these, they construct the ConceptGuard benchmark, in which forget sets and retain sets are explicitly complementary in concept usage. The benchmark uniquely enables unlearning to be explored and evaluated at the concept level rather than at a sparse fact level, with intent-sensitive evaluation aimed at maximizing contextual separation to promote safer behavior.

Key Findings

• Current unlearning techniques perform poorly under this context-sensitive setting, showing weak contextual separation.
• Results show poor ROUGE and concept-level metric performance.
• There is a strong unlearning–utility trade-off.
• Context sensitivity improves only marginally with existing methods.
• Concept-level control remains inconsistent across unlearning methods.

*Automatically collected on 2026-08-24*