Summary
A paper by Sahil Kale and Ian Harris (arXiv:2608.20338) introduces ConceptGuard, a new benchmark for evaluating context-sensitive unlearning in large language models. The authors argue that existing unlearning methods and benchmarks fall short: they rely on disjoint forget and retain sets composed of independent facts, and measure success with simple factual recall. This framing ignores a key requirement of unlearning—removing harmful behavior while preserving benign, useful knowledge. The paper proposes that unlearning should operate at the concept level, ensuring complete removal of unsafe applications while maintaining correct and beneficial usage. To formalize this, the authors introduce the notion of dual-use concepts—concepts that can be used both maliciously and benignly—and build ConceptGuard, where forget and retain sets are explicitly complementary in their concept usage. Experiments show that current unlearning techniques perform poorly in this setting, revealing a strong forgetting-utility trade-off and limited gains in context sensitivity.
Paper Overview
- Research area: NLP
- Authors: Sahil Kale, Ian Harris
- Published: 2026-08-22
- arXiv: 2608.20338
Abstract
Large language models face growing demand for selective knowledge removal, but existing methods and benchmarks cannot fully evaluate this capability. Current approaches rely on disjoint forget and retain sets composed of independent facts, using simple factual recall to measure success.
This framework overlooks a key requirement of unlearning: eliminating harmful behavior while preserving benign, useful knowledge. The paper argues that unlearning must operate at the concept level, ensuring complete removal of unsafe applications while maintaining their correct and useful usage.
To this end, the authors introduce dual-use concepts—concepts that can be used both harmfully and benignly—and construct the ConceptGuard benchmark, in which the forget and retain sets are explicitly complementary in concept usage.
Experiments demonstrate that current unlearning techniques perform poorly in this setting, revealing a strong forgetting-utility trade-off and limited improvements in context sensitivity.
*Auto-collected on 2026-08-22.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633806