Imagine a doctor prescribes you a drug with a very technical-sounding name. You would not question it — unless the drug simply does not exist. A recent paper, PhantomBench, reveals a disturbing pattern: when language models are asked about completely nonexistent concepts, they rarely say "I don't know." Instead, they confidently fabricate plausible explanations, with hallucination rates reaching 86.7% in some settings.
Building a Test Out of Things That Don't Exist
PhantomBench's core idea is elegant: to find out whether someone truly knows or is faking, ask them about something you are certain does not exist.
The research team built an automated pipeline to generate nonexistent concepts:
1. Decompose real concepts: break existing terms and entities into smaller components (words, n-grams). 2. Recombine: assemble these components into new, plausible-sounding but fabricated concepts. For example, combining components of "quantum mechanics" and "social Darwinism" yields a fake but convincing-sounding term. 3. Frequency filtering: search large-scale corpora to ensure the fabricated concepts genuinely have zero matches. 4. Diversified probing: ask from 7 different angles — does this concept exist? What does it mean? When was it coined? Where? What is its etymology? What are its applications? What is it related to?
The result: over 60,000 nonexistent terms and entities spanning science, law, medicine, history, and other domains.
Alarming Results
The team evaluated 21 models of different types and sizes and found:
No model reliably refuses to answer questions about nonexistent concepts. When the input presupposes that these concepts exist, models almost never challenge the premise — they just run with it.
Key findings:
- Prompt type matters enormously: asking "What does [fake concept] mean?" triggers hallucination far more easily than "Does [fake concept] exist?" Models tend to go along with the question's presupposition.
- Bigger models are not necessarily more reliable: more parameters do not automatically produce better "knowing what you don't know" behavior.
- Domain expertise is no guarantee: models that perform well in medicine still fabricate nonexistent medical terms.
- Reasoning models "reason from nothing": even chain-of-thought models will reason about nonexistent concepts rather than stop and question them.
Why This Is More Dangerous Than Ordinary Hallucination
Ordinary hallucinations — getting a real fact wrong — can at least be verified. PhantomBench exposes a subtler threat: when the concept itself does not exist, there are no facts available to refute the model's answer.
A user reading a fluent explanation of a fabricated concept has no way to judge whether the concept is real. It is like a con artist inventing a legal statute you have never heard of — you cannot disprove it, because you do not know it does not exist.
Nonexistent Concepts as Proxies for Rare Concepts
The paper has another clever extension: nonexistent concepts can serve as proxies for rare concepts when studying model behavior.
In the real world, models hallucinate more on low-frequency concepts (due to sparse training data). But rarity is hard to define and evaluate. Nonexistent concepts are naturally "the rarest," providing a clean experimental environment.
The study finds that model performance on nonexistent concepts correlates strongly with performance on genuinely rare concepts — meaning PhantomBench does not just measure hallucination, it can predict behavior near models' knowledge boundaries.
From "Can It Fabricate" to "Will It Say No"
This work points to a fundamental issue: an AI system's reliability depends not only on how much it knows, but on whether it knows what it does not know.
An honest model should say "I'm not sure this concept exists" for out-of-scope questions rather than fabricating plausible answers. Current models clearly fall short.
PhantomBench offers a simple but powerful evaluation framework: if you ask about a nonexistent concept, any answer other than "I'm not sure" counts as a hallucination. This binary judgment is far easier — and more reliable — than evaluating open-ended hallucination.
---
Paper: PhantomBench: Benchmarking the Non-existential Threat of Language Models Authors: Haeji Jung, Hila Gonen (University of British Columbia) Link: https://arxiv.org/abs/2606.11105