K-Bench: Exposing the Illusion of LLM 'Forgetting' in Agentic Deployments

Forum topic · 小凯 · 2026-09-14

Summary

A zhichai.net analysis of the paper 'K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments' (arXiv:2609.12808), which argues that existing unlearning benchmarks like TOFU and MUSE only probe a model's final answer, missing privacy leaks across other observable channels. K-Bench evaluates LLM unlearning through six channels of a ReAct agent: chain-of-thought text, tool-call arguments, tool returns, retrieval results, final answers, and post-hoc summaries. Testing 20 published unlearning methods (gradient ascent, gradient difference, DPO variants, LoKU, IDK fine-tuning, input corruption, activation editing, and others) on secrets hidden in four substrates—parameters, prompts, free-text retrieval, and structured retrieval—the authors find that when data lives in prompts or retrieval stores, single-channel benchmarks report no leakage while the same model leaks PII in 22%-86% of agent queries. Many methods merely suppress the answer channel (K-SUP channel suppression), shifting leakage to tool returns rather than removing it. No method achieved verifiable selective forgetting; input corruption was the only partial success, at a utility cost. The benchmark uses preregistered hypotheses, paired McNemar tests with Benjamini-Hochberg correction, and collapse-aware scoring, with findings robust across injection recipes, scoring rules, and base models.

K-Bench: Exposing the Illusion of LLM "Forgetting" in Agentic Deployments

> *"Sweeping dust under the rug makes the floor look clean. But the dust hasn't vanished—it just moved."*

Paper overview

Title: K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments
Authors: Guangsheng Yu, Yanna Jiang, Qin Wang, Baihe Ma, Xu Wang
Institutions: University of Technology Sydney, CSIRO
arXiv: 2609.12808

The blind spot in existing benchmarks

Machine unlearning aims to remove specific knowledge (e.g., personal data protected under GDPR's "right to be forgotten") from trained models. Leading benchmarks TOFU (Maini et al., 2024) and MUSE (Shi et al., 2025) verify forgetting by probing only the final answer—direct questions and prefix completion.

This creates a fatal blind spot: a model can output "I don't know" and score perfectly while the secret remains intact in its weights—unable to distinguish "truly forgot" from "remembers but refuses to say."

The six windows of K-Bench

K-Bench wraps the model in a ReAct agent and observes six channels. Leakage is flagged by logical OR across all of them:

| Channel | Content | |---|---| | Z_CoT | Chain-of-thought reasoning text | | Z_tool | Tool-call arguments | | Z_tool_wide | Tool arguments plus tool return values | | Z_RAG | Retrieved documents | | Z_answer | Final answer | | Z_summary | Post-hoc summary under follow-up questioning |

A textbook case of channel migration

Testing StaR (reasoning-chain filtering) on Llama-3.1-8B with a secret birth date (1999-03-17) stored in a structured retrieval database:

Single-channel evaluation sees the answer perturbed to "1999-03-27" → judged as *successfully forgotten*.
But K-Bench's wider view reveals the tool-call arguments hit the record exactly, and the tool return contains the original date: 1999-03-17.
Aggregate OR(all) leakage = 0.857, essentially identical to the no-intervention baseline (0.855).

The authors call this K-SUP (channel suppression): the dominant leakage channel changed, but total leakage did not meaningfully drop.

Where the secret hides determines where it leaks

K-Bench injects secrets into four substrates:

1. P (parameters) — baked into weights via training/fine-tuning 2. C (context) — inside prompts or injected records 3. R-text — free-text retrieval corpora 4. R-struct — structured databases queried by fields

Weight-editing methods can only touch P; inference-time interventions touch all input sides but not weights; retrieval-side methods only affect R. Parameter-only benchmarks like TOFU and MUSE are completely blind to C and R—yet deployed agents routinely handle PII through prompts and retrieval stores.

The headline number: when secrets live in prompts or retrieval stores, TOFU/MUSE report zero leakage, while the same model deployed as an agent leaks the target PII in 22%–86% of queries.

Judgment day for 20 unlearning methods

The authors evaluate 20 published weight-based methods (GA, GU, DPO variants, LoKU, IDK, NPO, W2SF, RLF, R2S, VLMU, DAM, SalUn, SCRUB, and others) on their "home turf"—the parameter substrate—under multi-channel observation:

Most methods silence the answer channel, but the secret can still be elicited from side channels, especially the follow-up summary channel.
Some methods collapse the model entirely—TOFU-style scoring even counts collapse as "success" since the answer channel stops leaking.
IDK (refusal fine-tuning) teaches the model to say "I don't know"—learning to lie rather than deleting memory; secrets still seep out under strong elicitation.
Activation editing even triggered agent crashes.

No published method achieved verifiable selective forgetting under the multi-channel observer. The only approach meeting the standard was the crudest: input corruption, at the cost of input-side utility.

Notably, changing the base model (Llama-3.1-8B vs Qwen vs Mistral) changes which method ranks first—single-model leaderboards are unlikely to generalize.

Making benchmarks scientific

K-Bench applies scientific rigor: preregistered hypotheses, seed-pooled paired McNemar tests, Benjamini–Hochberg false-discovery-rate correction, collapse-aware K-Score, and per-channel failure localization. Robustness checks confirm the "all 20 methods fail" finding is insensitive to injection recipes, scoring rules (exact vs semantic matching), question formats, and run-to-run variance.

The authors honestly state boundaries: single-agent only (multi-agent systems expose a strict superset of channels), one substrate per experiment, and OR(all) detection misses fragmentation attacks that reconstruct secrets by combining channel fragments—so reported leakage is a lower bound.

Conclusion

The paper's core message: silence on the answer channel is not evidence of forgetting—it just means the secret learned to keep quiet. In the agentic era, "forgetting" must be redefined as "the secret cannot be recovered on any observable surface the agent exposes." The six windows have now been counted; the task ahead is closing each one properly, not sweeping more dust under the rug.

References

1. Yu G., et al. "K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments." arXiv:2609.12808. 2. Maini P., et al. "TOFU: A task of fictitious unlearning for LLMs." arXiv:2401.06121, 2024. 3. Shi W., et al. "MUSE: Machine unlearning six-way evaluation." arXiv:2503.02657, 2025. 4. Yao S., et al. "ReAct: Synergizing reasoning and acting in language models." ICLR, 2023.