Imagine a doctor prescribes you a drug called 'fluoxetine-triazolone.' You wouldn't question it — the name sounds professional. But what if the drug simply doesn't exist?
A new paper, PhantomBench, reveals a troubling fact: when language models are asked about concepts that do not exist at all, they don't say "I don't know" — they confidently fabricate plausible explanations. In some settings, the hallucination rate reaches 86.7%.
Building a Test Out of Things That Don't Exist
PhantomBench's core idea is elegant: if you want to know whether someone truly understands or is bluffing, ask them about something you are certain does not exist.
The researchers built an automated pipeline to generate "non-existent concepts":
1. Decompose real concepts: break existing terms and entities into smaller components (words, n-grams) 2. Recombine: assemble these components into new, plausible-sounding but fabricated concepts. For example, recombining pieces of "quantum mechanics" and "social Darwinism" yields "quantum social mechanics" — impressive-sounding, but entirely made up 3. Frequency filtering: search large-scale corpora to ensure the generated concepts have zero matches 4. Diverse probing: ask from 7 different angles — does this concept exist? What does it mean? When was it coined? Where? What is its etymology? What are its applications? What is it related to?
The pipeline produced over 60,000 non-existent terms and entities spanning science, law, medicine, history, and more.
Alarming Results
The team evaluated 21 models of different types and scales and found:
No model reliably refuses to answer questions about non-existent concepts. When the input presupposes these concepts exist, models almost never question the premise — they go along and fabricate.
Key findings:
- Prompt phrasing matters enormously: asking "What does quantum social mechanics mean?" triggers hallucination far more than asking "Does quantum social mechanics exist?" Models tend to accommodate the presuppositions in the prompt
- Bigger models are not necessarily more reliable: more parameters do not automatically improve "knowing what you don't know"
- Domain expertise doesn't guarantee reliability: models that perform well in medicine still fabricate non-existent medical terms
- Reasoning models "reason about nothing": even chain-of-thought reasoning models will "reason" about non-existent concepts rather than stop and question them
Why This Is More Dangerous Than Ordinary Hallucination
Ordinary hallucinations — getting a real fact wrong — can at least be fact-checked. PhantomBench exposes a stealthier threat: when the concept itself doesn't exist, there is no fact to refute the model's answer against.
A user reading a fluent explanation of "quantum social mechanics" has no way to judge whether the concept is real. It's like a con artist citing a legal statute you've never heard of — you can't falsify it, because you don't know it doesn't exist.
Non-Existent Concepts as a Proxy for Rare Concepts
The paper also offers a clever extension: non-existent concepts can serve as proxies for rare concepts when studying model behavior.
In the real world, models hallucinate more on low-frequency concepts (due to scarce training data). But rarity is hard to define and evaluate — how do you measure how "rare" a concept is? Non-existent concepts are naturally "the rarest," providing a clean experimental setting.
The study finds that model performance on non-existent concepts is highly correlated with performance on genuinely rare concepts. This means PhantomBench doesn't just measure hallucination — it can predict model behavior near the boundary of its knowledge.
From "Can It Fabricate" to "Will It Say No"
This work points to a fundamental question: the reliability of AI systems depends not only on how much they know, but on how well they know what they don't know.
An honest model should say "I'm not sure this concept exists" when asked about something beyond its knowledge, rather than fabricating a plausible answer. Current models clearly fail at this.
PhantomBench provides a simple but powerful evaluation framework: ask about a non-existent concept, and any answer other than "I'm not sure" is a hallucination. This binary judgment is far easier — and more reliable — than evaluating open-ended hallucination.
---
Paper: PhantomBench: Benchmarking the Non-existential Threat of Language Models Authors: Haeji Jung, Hila Gonen (University of British Columbia) Link: https://arxiv.org/abs/2606.11105