"Forgetting" Makes Models More Reliable: Erase Entity Names Before Training, and Hallucinations Drop
Have you ever asked ChatGPT a factual question and received a confident answer—only to check and find it wrong? Ask again with different phrasing, and it confidently gives another wrong answer. This "confident nonsense" is the well-known hallucination problem.
The root cause: LLMs pack massive facts into parameters during pretraining, but those facts may be outdated, incomplete, or contradictory. The model can't distinguish "I know" from "I'm guessing."
In July 2026, a research team from HPI, the University of Cape Town, the University of Copenhagen, and others proposed a seemingly radical but clever solution in the paper *Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling*: anonymize all named entities in the pretraining corpus, so the model fundamentally cannot learn specific facts and must learn to find answers from context.
They call this model KLLM — Knowledge-"Less" Language Model.
1. The Core Distinction: Parametric Recall vs. Evidence-Grounded Reasoning
Parametric recall: the model encodes "Paris is the capital of France" in its weights. Ask "What is the capital of France?" and it recalls from parameters. Fast, but unreliable — you don't know when the fact was encoded, whether it's accurate, or whether it conflicts with context.
Evidence-grounded reasoning: the model sees "Paris is the capital of France" in the context and answers accordingly. Slower, but traceable — the answer comes from provided material, not residual memory.
An ideal assistant should prioritize evidence-based reasoning. In reality, models over-rely on parametric recall: even given correct context, they can be misled by stale internal knowledge.
2. The KLLM Approach: Replace Names with Placeholders
The method is straightforward: using named entity recognition (NER), replace all named entities — person names, places, organizations, dates, numbers — with unified placeholders during pretraining.
Original: > "In 2023, OpenAI released GPT-4, scoring 86.4 on MMLU."
Anonymized: > "In [ENTITY_1], [ENTITY_2] released [ENTITY_3], scoring [ENTITY_5] on [ENTITY_4]."
The model learns grammar, reasoning patterns, and contextual relationships, but not that "OpenAI released GPT-4." It's like teaching someone chess rules and strategy without any historical games — they understand the principles but can't recite openings; show them the game record when needed.
3. Experimental Design: Same Architecture, Data, and Pipeline
For fair comparison, the team trained:
- SLM (Standard Language Model): normal pretraining, as baseline
- KLLM: same corpus, but with entities anonymized
- SmolLM-1.7B (2.5B tokens): SLM 55.9 → KLLM 59.8
- SmolLM-1.7B (10B tokens): SLM 47.7 → KLLM 53.8
- SmolLM-1.7B (2.5B tokens): SLM 16.3 → KLLM 19.2
- LLaMA-3B (2.5B tokens): SLM 22.1 → KLLM 26.9
- SmolLM-1.7B (2.5B tokens): SLM 64.3 → KLLM 74.7
- LLaMA-3B (2.5B tokens): SLM 67.7 → KLLM 75.8
- Closed-book QA collapses: without context, KLLM can barely answer factual questions — by design, but it limits use cases.
- Requires a retrieval system: KLLM must pair with BM25, DPR, or tool calling; it can't run "naked." This adds system complexity.
- Anonymization noise: imperfect NER may miss entities or mislabel non-entities; ablations show limited impact.
- Scale limits: only validated on models ≤1.7B; whether the trend holds at larger scale is unknown.
- RAG systems: higher faithfulness to retrieved evidence
- Enterprise knowledge bases: internal docs update frequently; parametric memory inevitably goes stale, KLLM reads only current documents
- High-risk domains (legal, medical): answers traceable to specific documents
- Multilingual settings: anonymization generalizes across languages, focusing capacity on cross-lingual reasoning
Both used identical architectures (SmolLM-135M/360M/1.7B, LLaMA-1B/3B), identical token counts (2.5B, 10B, 20B), and identical optimizer configurations. Evaluation covered five tasks: LAMA (fact recall), SQuAD (reading comprehension), NQ (natural questions), FEVER (fact verification), and HaluBench (hallucination detection).
4. Results: The Amnesiac Model Wins on Evidence Tasks
One sentence summarizes the results: KLLM is worse at parametric recall, but better on every task requiring contextual evidence.
Reading comprehension (SQuAD, with context):
Fact verification (FEVER):
Hallucination detection (HaluBench):
Interestingly, on LAMA (closed-book), KLLM sometimes scored *higher* (SmolLM-1.7B, 2.5B tokens: 48.5 vs 43.2) — the authors explain many LAMA items can be answered via reasoning, which KLLM performs more "purely" without factual memorization interfering.
The standout number: in retrieval-grounded settings with imperfect retrieved evidence, KLLM shows 20-25% relative improvement over SLM.
5. Why Does "Amnesia" Improve the Model?
1. Eliminates conflict between parametric recall and contextual evidence. Standard models activate parametric knowledge alongside context; if that knowledge is stale or wrong, it misleads. KLLM has nothing to activate and stays faithful to context. 2. Forces the model to "read" rather than "memorize." Anonymized training builds stronger comprehension from the start — like a student who never memorizes answers and must learn problem-solving. 3. Better calibration. KLLM beats SLM on ECE, Brier score, and AUROC. When it says "90% confident," it's actually right about 90% of the time; SLM's 90% confidence may correspond to only ~70% accuracy. 4. More reliable abstention. KLLM prefers saying "I don't know" or "not in the context" over fabricating answers.
6. The Costs
7. A Deeper Question: What Does "Knowing" Mean?
Traditionally, a model "knows" a fact if it's encoded in parameters. KLLM challenges this: perhaps "knowing" should be redefined as "being able to derive from evidence" rather than "memorized in weights."
There's a human parallel: a historian may not remember every date, but knows how to look things up, cross-verify, and judge source credibility. Their knowledge lies in method, not recitation.
More radically, this may be a fundamental direction against hallucination. Rather than using RLHF after training to teach a model not to hallucinate (treating symptoms), change the training paradigm so the model *cannot* fabricate — it simply doesn't know specifics, only what the context says.
8. Practical Applications
9. Limitations and Outlook
The authors candidly list: unverified scale (only ≤1.7B tested), dependence on NER quality, dependence on retrieval quality, and incomplete mechanistic understanding of why anonymization improves calibration.
Still, as a proof of concept, KLLM is persuasive. It proposes a different pretraining paradigm rather than a tweak — a rare breath of fresh air in today's "bigger model, more data" arms race.
10. Conclusion
"Amnesia" improving models sounds paradoxical, but makes sense on reflection. Human knowledge has never relied on one person memorizing everything — it relies on knowing where to look and how to judge what you find. In an era of ever-fuller parameters and ever-subtler hallucinations, this paper reminds us: sometimes knowing fewer specific facts but more reliable methods is the safer path.
---
Paper: https://arxiv.org/abs/2607.12831 HTML full text: https://arxiv.org/html/2607.12831