ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Forum topic · 小凯 · 2026-08-24

Summary

ConceptGuard is a new benchmark by Sahil Kale and Ian Harris (arXiv:2608.20338) that evaluates context-sensitive machine unlearning in large language models. The authors argue that effective unlearning must operate at the concept level: harmful applications of knowledge should be removed while benign, beneficial uses remain intact. To test this, they introduce the notion of dual-use concepts—knowledge that can be applied both harmfully and benignly—and build a benchmark whose forget and retain sets are explicitly complementary in concept usage. Evaluation is intent-sensitive and targets maximizing contextual separation to promote safer model behavior. Experiments show that current unlearning techniques perform poorly in this setting, exhibiting weak contextual separation, low ROUGE and concept-level metric scores, a strong unlearning-utility trade-off, and inconsistent concept-level control across methods. The paper highlights a gap in existing unlearning benchmarks, which rely on disjoint fact sets and simple fact recall, and positions ConceptGuard as a more meaningful standard for evaluating safe, concept-level knowledge removal in LLMs.

Paper Overview

Field: NLP
Authors: Sahil Kale, Ian Harris
Published: 2026-08-22
arXiv: 2608.20338

Abstract

Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and benchmarks fail to evaluate this capability completely. Current approaches rely on disjoint forget and retain sets composed of independent facts, and measure success with simple direct fact recall. This framing misses a key requirement of unlearning: the ability to eliminate harmful behaviors while preserving benign, beneficial knowledge.

The authors argue that effective unlearning must operate at the concept level, ensuring full removal of unsafe applications while retaining their correct and useful uses—achieving a complete and meaningful unlearning in a conceptual sense.