ConceptGuard: Benchmarking Context-Sensitive Unlearning in LLMs

Forum topic · 小凯 · 2026-08-22

Summary

ConceptGuard is a benchmark by Sahil Kale and Ian Harris (arXiv:2608.20338) that evaluates context-sensitive machine unlearning in large language models. The authors argue that existing unlearning methods and benchmarks are inadequate because they rely on disjoint forget and retain sets composed of independent facts, measuring success through simple factual recall. This framing overlooks a key requirement of unlearning: eliminating harmful behavior while preserving benign, beneficial knowledge. The paper proposes that unlearning must operate at the concept level, ensuring complete removal of unsafe applications while maintaining correct and useful usage. To this end, the authors introduce the notion of dual-use concepts—knowledge that can be applied both harmlessly and harmfully—and build ConceptGuard, where forget and retain sets are explicitly complementary in concept usage. Experiments show that current unlearning techniques perform poorly in this setting, revealing a strong forgetting-utility trade-off and only limited improvements in contextual sensitivity. The work highlights a significant gap in today's unlearning research and provides a more realistic evaluation standard for selective knowledge removal in LLMs.

Paper Overview

Field: NLP Authors: Sahil Kale, Ian Harris Published: 2026-08-22 arXiv: 2608.20338

Abstract

Large language models face growing demands for selective knowledge removal, but existing methods and benchmarks fail to fully evaluate this capability. Current approaches rely on disjoint forget and retain sets made up of independent facts, using simple factual recall to measure success. This framing ignores a critical requirement of unlearning: eliminating harmful behavior while preserving benign, beneficial knowledge.

This paper proposes that unlearning must operate at the concept level, ensuring complete removal of unsafe applications while maintaining correct and useful usage. To this end, the authors introduce dual-use concepts—concepts that can be used both harmfully and benignly—and construct the ConceptGuard benchmark, in which the forget and retain sets are explicitly complementary in concept usage.

Key Findings

Existing unlearning techniques perform poorly on ConceptGuard's concept-level setting.
Results reveal a strong forgetting-utility trade-off: removing harmful usage degrades benign usage.
Current methods show only limited gains in contextual sensitivity—the ability to distinguish harmful from benign applications of the same concept.

Significance

ConceptGuard exposes a fundamental gap in current unlearning research, shifting evaluation from simple fact removal to context-aware selective forgetting, and provides a more realistic standard for assessing safety-focused knowledge removal in LLMs.

--- *Auto-collected on 2026-08-22*