Smarter LLMs Suffer More from Anonymization: A Counterintuitive Finding
> On the Impact of Anonymization on the Performance of Large Language Models > arXiv: 2609.11335 | Deußer, Hahnbück, Sparrenberg, Uelwer et al. > Five mainstream LLMs × eleven benchmarks × original vs. anonymized inputs
---
A Result That Makes You Stop
Suppose you are building a medical QA system. For privacy compliance, you replace all person names, locations, and organization names in the input with placeholders like [PERSON_1], [ORG_2]. Intuitively, this shouldn't matter much—models at the level of GPT-4o should understand that "Patient A is allergic to drug B" and "Zhang San is allergic to penicillin" mean the same thing.
Wrong.
The paper finds that the stronger the model, the larger the performance drop caused by anonymization. Capable models like Qwen2.5-72B and GPT-4o mini fall hardest after anonymization, while smaller models are less affected.
It's like an experienced physician who habitually asks "Where do you work? Does anyone in your family have similar symptoms?"—suddenly stripping that context away leaves them more lost than an intern.
---
Experimental Setup: Five Models × Eleven Benchmarks
The researchers selected five mainstream language models:
- Qwen2.5-72B (large model representative)
- GPT-4o mini (commercial closed-source representative)
- Three smaller open-source models
[PERSON_1], preserving entity uniqueness) and irreversible anonymization (direct deletion or [REDACTED] substitution).---
Three Counterintuitive Findings
Finding 1: The Stronger the Model, the Harder the Fall
Anonymization's impact is not uniform. Qwen2.5-72B and GPT-4o mini show the clearest performance degradation after anonymization. The authors speculate that stronger models learn to exploit entity information more deeply during training—they don't treat "Zhang San" as just a symbol, but associate it with a whole web of contextual knowledge. Once entities are erased, those connections all break.
This reveals an overlooked phenomenon: part of a model's capability comes from "memorizing" implicit relationships between entities, not from purely abstract reasoning. The stronger the capability, the deeper the dependence.
Finding 2: TruthfulQA Actually Improved After Anonymization
The most puzzling result in the study. TruthfulQA tests whether models propagate false common sense—e.g., "Do vaccines cause autism?" After anonymization, model performance on this benchmark improved.
The researchers' explanation: many of TruthfulQA's trap questions rely on specific entities (e.g., false claims attributed to specific people) to induce the model. Anonymization removed these entity cues, preventing the model from being led astray and pushing it back toward more general factual judgment.
This suggests that some of the model's "errors" on TruthfulQA are triggered by specific entities—seeing a certain name reflexively prompts related misinformation. Anonymization accidentally severed that reflex.
Finding 3: Retrieval Tasks Suffered "Catastrophic Decline"
In sharp contrast to TruthfulQA's improvement, retrieval-grounded tasks like RGB—answering questions based on input documents—showed catastrophic degradation.
The reason is direct: the core of retrieval tasks is "find entity X mentioned in the document, then answer questions about X." After anonymization, "Zhang San" in the document becomes [PERSON_1], and "Zhang San" in the question also becomes [PERSON_1]—but the model can no longer link the two, because the placeholders lost the entities' semantic information.
---
Two Fixes: One Works, One Doesn't
The researchers also tested two potential remedies:
Option 1: Reversible vs. irreversible anonymization.
Reversible anonymization (preserving entity uniqueness, e.g., [PERSON_1], [PERSON_2]) significantly outperformed irreversible anonymization (deletion or [REDACTED]). This shows models can use the information "these are two different entities" even without knowing who they are.
Option 2: Explicitly telling the model the input is anonymized.
The researchers tried explicitly stating in the prompt: "the following text has been anonymized; [PERSON_1] represents a specific person's name." Result: no discernible benefit. The model knows it is facing anonymized input, but cannot compensate for the loss.
This matters: it shows anonymization's impact is not a surface-level issue of "the model is confused about what placeholders mean," but something deeper—the model's internal reasoning pathways that depend on entity semantics are severed, and prompt-level hints cannot repair that.
---
What This Means
The paper's core insight in one sentence: anonymization is not a one-click switch, but an engineering problem that must be co-designed with the model and the task.
For domains that must protect privacy—healthcare, law, finance—this means: 1. Don't assume large models are "immune" to anonymization—on the contrary, the stronger the model, the larger the potential performance loss 2. Reversible anonymization is the better choice—preserve entity uniqueness even without preserving identity 3. Be especially careful with retrieval tasks—they may fail outright under anonymization 4. Fact-checking tasks may unexpectedly benefit—TruthfulQA's improvement hints that some misinformation propagation is entity-triggered
---
A Deeper Question
The paper leaves an open question worth pondering: if stronger models depend more on entity information, is that dependence "true intelligence" or "sophisticated pattern matching"?
A model that truly understands "Patient A is allergic to drug B" shouldn't change its judgment based on whether A is "Zhang San" or [PERSON_1]. But the experiments show that stronger models rely more on this concrete information. This may indicate that much of current LLMs' "capability" is built on remembering entity relationships rather than genuine abstract reasoning.
Anonymization inadvertently became a probe: it measured the ratio of "entity-dependent" to "abstraction-dependent" capability within models. The result—the entity-dependent portion is far larger than we assumed.
---
Paper link: arxiv.org/abs/2609.11335
Code: No public code repository provided by the authors.