"Amnesia" Makes Models More Reliable: Erasing Entity Names During Pretraining Reduces Hallucination
Have you ever asked ChatGPT a factual question, received a confident answer, checked it, and found it wrong? Ask again with different phrasing, and it confidently gives you another wrong answer.
This "confident nonsense" is the well-known hallucination problem. The root cause: large models cram vast amounts of factual knowledge into their parameters during pretraining, but this knowledge may be outdated, incomplete, or contradictory. The model cannot distinguish "I know" from "I'm guessing."
In July 2026, a research team from HPI, the University of Cape Town, the University of Copenhagen, and others proposed a seemingly radical but actually elegant solution in the paper *Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling*: anonymize all named entities in the corpus before pretraining, so the model never learns specific facts in the first place and can only learn to find answers from context.
They call this model KLLM — Knowledge-"Less" Language Model.
1. The Core Problem: Parametric Recall vs Evidence-Grounded Reasoning
Two concepts first:
Parametric recall: the model encodes "Paris is the capital of France" in its weights. Ask "What is the capital of France?" and it "recalls" the answer from parameters. Fast, but unreliable — you don't know when the fact was encoded, whether it's accurate, or whether it conflicts with the context.
Evidence-grounded reasoning: the model sees "Paris is the capital of France" in the context and answers accordingly. Slightly slower, but traceable — the answer comes from the material you provided, not residual memory in the model's head.
Ideally, a reliable assistant should prioritize evidence-grounded reasoning. In reality, models rely too heavily on parametric recall — even given correct context, they can be misled by stale parametric knowledge.
2. How KLLM Works: Replacing Names with Placeholders
The approach is direct: using named entity recognition (NER), all named entities in the pretraining corpus — person names, places, organizations, dates, numbers — are replaced with uniform placeholders.
Original text: > "In 2023, OpenAI released GPT-4, scoring 86.4 on MMLU."
After anonymization: > "In [ENTITY_1], [ENTITY_2] released [ENTITY_3], scoring [ENTITY_5] on [ENTITY_4]."
The model only learns grammar, reasoning patterns, and contextual relationships — not the fact that "OpenAI released GPT-4."
It's like teaching someone chess: you teach rules and strategy but no historical games. They can't memorize openings, but they understand the game. To review a specific game, you hand them the record.
3. Experimental Design: Same Architecture, Same Data, Same Pipeline
For fair comparison, the team trained two groups:
- SLM (Standard Language Model): normal pretraining, as baseline
- KLLM: pretrained on the same corpus, but with entities anonymized
- SmolLM-1.7B (2.5B tokens): SLM 43.2 → KLLM 48.5 (KLLM higher? The authors explain many LAMA questions can be answered via reasoning; KLLM's reasoning is "purer," uncontaminated by memorized facts)
- SmolLM-1.7B (2.5B tokens): SLM 55.9 → KLLM 59.8
- SmolLM-1.7B (10B tokens): SLM 47.7 → KLLM 53.8
- SmolLM-1.7B (2.5B tokens): SLM 16.3 → KLLM 19.2
- LLaMA-3B (2.5B tokens): SLM 22.1 → KLLM 26.9
- SmolLM-1.7B (2.5B tokens): SLM 64.3 → KLLM 74.7
- LLaMA-3B (2.5B tokens): SLM 67.7 → KLLM 75.8
- Closed-book QA degrades dramatically. Without context, KLLM can barely answer factual questions — by design, but it limits use cases.
- Requires a retrieval system. KLLM must be paired with retrieval (BM25, DPR) or tool calling; it cannot run "naked." This adds system complexity.
- Noise from anonymization. NER is imperfect — some entities are missed, some non-entities mislabeled. Ablation studies in the appendix show minimal impact on final results.
- Scale limitations. Validated only up to 1.7B parameters; whether the trend holds at larger scale is unknown.
- RAG (retrieval-augmented generation) systems: KLLM is more faithful to retrieved evidence.
- Enterprise knowledge base Q&A: internal documents change constantly; parametric recall inevitably goes stale. KLLM only reads current documents, naturally avoiding "answering new questions with outdated knowledge."
- High-risk domains like law and medicine: hallucination is intolerable; answers must be traceable to specific documents. KLLM's "evidence-first" nature fits naturally.
- Multilingual settings: anonymization generalizes across languages, letting KLLM focus on cross-lingual reasoning.
- Scale unverified: tested only below 1.7B; unknown whether the trend holds at tens of billions of parameters
- Imperfect entity recognition: anonymization quality is limited by NER accuracy
- Dependence on external retrieval: poor retrieval degrades KLLM too
- Mechanism not fully understood: why anonymized training improves calibration is only observed empirically
Both groups used the same architectures (SmolLM-135M/360M/1.7B, LLaMA-1B/3B), the same token counts (2.5B, 10B, 20B), and identical optimizer configurations. The only difference: entities present or not.
Evaluation covered 5 tasks: LAMA (fact recall), SQuAD (reading comprehension), NQ (natural questions), FEVER (fact verification), HaluBench (hallucination detection).
4. Results: The Amnesiac Model Performs Better
One sentence summarizes the results: KLLM got worse at tasks requiring parametric recall, but better at every task requiring contextual evidence.
Fact recall (LAMA, closed-book):
Reading comprehension (SQuAD, with context):
Fact verification (FEVER):
Hallucination detection (HaluBench):
Most striking number: in retrieval-grounded settings, when retrieved evidence is imperfect, KLLM achieves a 20-25% relative improvement over SLM.
5. Why Does "Amnesia" Improve the Model?
Intuition says knowing more facts should help. Why does removing facts improve performance?
1. It eliminates conflicts between parametric recall and contextual evidence. Standard models activate parametric knowledge when reading context. If the two agree, fine; if parametric knowledge is outdated or wrong, the model gets misled. KLLM has nothing to activate — it can only rely on context, and is therefore more faithful.
2. It forces the model to "read" rather than "memorize." Anonymized training makes the model understand entity relationships through context from the start, cultivating stronger reading comprehension. Like a student who never memorizes answers and must learn problem-solving methods.
3. Better calibration. KLLM outperforms SLM on ECE, Brier score, and AUROC. When it says "I'm 90% sure," it's actually right about 90% of the time; an SLM saying "90% sure" may only be right 70% of the time. This is crucial for deployment — a model that knows what it doesn't know is far more useful than one that is confident about everything.
4. More reliable abstention. KLLM tends to say "I don't know" or "this isn't in the context" when uncertain, rather than fabricating an answer.
6. What Are the Costs?
KLLM is not free:
7. A Deeper Implication: What Is Knowledge?
This paper raises a philosophical question: for a language model, what does it mean to "know" a fact?
The traditional answer: encoded in parameters. KLLM challenges this. It suggests that "knowing" should be redefined as "being able to derive it from evidence" rather than "memorized in weights."
There's a subtle parallel with human cognition. A historian may not remember every date, but knows how to look things up, cross-verify, and judge source reliability. Their knowledge lies in method, not memorization. KLLM follows this path.
More radically, this could be a fundamental direction for solving hallucination. Rather than using RLHF after training to "teach" models not to hallucinate (like teaching a liar not to lie — treating symptoms, not causes), change the training paradigm so the model *can't* lie — because it simply doesn't know specific facts and can only say what's in the context.
8. Practical Applications
KLLM best suits:
9. Limitations and Outlook
The authors honestly list several limitations:
10. Conclusion
"Amnesia" making models better sounds paradoxical, but on reflection it makes sense. Human knowledge has never depended on one person memorizing everything — it relies on "knowing where to look" and "judging whether what you find is trustworthy." KLLM applies the same wisdom to AI.
In an era where models cram in more parameters while hallucinations grow subtler, this paper reminds us: sometimes, knowing fewer specific facts while learning better methods is the more reliable path.
---
Paper: https://arxiv.org/abs/2607.12831 HTML full text: https://arxiv.org/html/2607.12831