A New Version of an Old Fable
In the parable of the blind men and the elephant, each man touches a different part — a leg, a trunk, an ear — and each concludes the elephant is something different. Everyone is partially right, and everyone misses the rest.
A 2026 paper applies this parable to LLMs: Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge.
Core finding: even the strongest frontier models, when facing multiple divergent accounts of the same fact, recall both accounts simultaneously only 52.4% of the time. For nearly half of the questions, models remember only the mainstream version and completely forget the minority account.
The LLM doesn't lack self-knowledge of what it knows — it knows the mainstream version, but doesn't know what it's missing.
ElephantBench: 1,094 Divergent Questions
The paper builds ElephantBench, a benchmark of 1,094 questions where different sources give different accounts of the same fact.
Example:
- Source A (IMDb): "Mother Teresa was born on August 26, 1910 ... born Agnes Gonxha Bojaxhiu."
- Source B (Poem of Quotes): "Mother Teresa, Agnes Gonxha Bojaxhiu, born August 27, 1910 in Skopje, Macedonia."
- Question: When was Mother Teresa born?
- Best model: probability of recalling both accounts = 52.4%
- Average model: ≈ 39.4%
- Recalling only the mainstream account: ≈ 23.2% of model-question pairs
- Recalling neither account: ≈ 2.19%
- Higher mainstream exposure → tendency to recall only the mainstream version
- Higher minority exposure → tendency to recall both versions
- Evaluation blind spot: traditional QA can't detect whether models remember divergent accounts. ElephantBench exposes a new completeness blind spot — a model can answer the mainstream version correctly while entirely omitting the minority version.
- Judgment-gate decoupling: the fix isn't making models "smarter" but making output mechanisms preserve diversity — repairing gates is easier than repairing judgment.
- Scalar illusion: QA accuracy is a scalar, but model knowledge is a vector (which versions exist, their strengths, their relations). Managing a vector with a scalar is like measuring blood pressure with a thermometer.
- New concept — epistemic myopia: a systematic, data-structure-driven bias toward recalling mainstream accounts, applicable to all frequency-trained systems including search engines and recommender systems.
- Limited scale: 1,094 questions vs. MMLU's 14,000
- Long-tail definition depends on \(D_{low}\): different corpus choices could yield different divergence questions
- Limited ecological validity: real systems rarely answer fact questions fully closed-book
- Model recency: evaluated models date to early 2026; newer frontier models may perform better
Both sources are human-verified and credible, yet they give different dates (Aug 26 vs Aug 27).
Evaluation method: closed-book. The model cannot view the source texts; it must recall from parametric memory. What's tested is not "can the model answer" but "can the model recall both divergent accounts simultaneously."
This differs fundamentally from traditional QA evaluation, which assumes one correct answer per question. ElephantBench assumes multiple correct answers may exist — the key is not answering *correctly* but answering *completely*.
Construction Pipeline: "Tracing the Elephant"
1. Mine long-tail corpora — retrieve documents from low-exposure web corpora \(D_{low}\), typically absent from mainstream training data. 2. Build a document graph — identify document pairs describing the same fact with different accounts; edges represent divergence. 3. Generate questions — one closed-book question per divergent pair, answerable from both sources but with different values. 4. Human review — annotators verify that divergences are real and questions unambiguous. 5. Closed-book evaluation — models recall purely from parametric memory.
The pipeline's cleverness: it redefines "long-tail" from "rare" to "divergent." Traditional long-tail evaluation tests whether models know rare facts; ElephantBench tests whether models can simultaneously remember multiple versions of the same fact.
Epistemic Myopia Across 32 Models
Key numbers
52.4% is the ceiling. Even frontier models capture both versions only slightly more than half the time.
Scale effect
Larger models recall more completely — but incompleteness does not vanish. From 7B to 70B to frontier models, full recall rises but stays in the 50–60% range.
Reasoning mode effect
Chain-of-thought outperforms direct answering — but incompleteness persists. Even given time to "think," models still drop minority accounts.
Open vs. closed models
A significant gap exists between open models (Llama, Qwen) and closed models (GPT, Claude). Closed models are more complete, but open models are close on mainstream-account recall — the gap concentrates on minority accounts.
Exposure Imbalance: The Root Cause
The paper's deepest finding is the exposure imbalance effect. In training corpora, different accounts of the same fact appear at different frequencies (e.g., one birth date appears 1,000 times, the other 10 times). This maps directly onto model recall:
This explains why neither scale nor reasoning eliminates myopia: the problem lies not in model capability but in the structure of training data. If the data is imbalanced, the model learns an imbalanced distribution — an information-theoretic version of "garbage in, garbage out": not a bad model, incomplete data.
Analogy: Survivorship Bias in Memory
In WWII, analysts examined returning bombers and found bullet holes concentrated on the wings. The intuitive fix: armor the wings. But Abraham Wald pointed out that the tail holes were missing not because tails were rarely hit, but because tail-hit planes never returned — you see the survivors, not the non-survivors.
LLM parametric memory works the same way. The recalled mainstream account "survived" on the high-frequency side of training data; minority accounts are the non-survivors — present in the data but drowned by high-frequency mainstream versions. The model doesn't lack knowledge of the minority; the mainstream account's activation strength suppresses it. This mirrors a "judgment-gate decoupling": the model may internally encode both accounts (judgment), but the output mechanism only releases the mainstream one (gate).
Connections to Broader Concepts
Practical Implications
1. Closed-book evaluation has limits. Real applications often use open-book settings (RAG, search augmentation). The paper doesn't test this directly, but exposure imbalance implies: even open-book, if retrieval results are imbalanced, myopia persists. 2. Completeness matters more than correctness. With divergent facts, "correct" is ill-defined — both sources are right, and models should present both. Alignment should target "output the complete distribution," a harder goal. 3. Long-tail means "divergent," not just "rare." Many facts aren't black-and-white but have multiple plausible versions. 4. PPL as an efficient proxy. Perplexity on divergent documents can approximate recall completeness without running the full 1,094-question evaluation.
Honest Assessment
Limitations:
Conclusion
The parable says everyone sees only the part they touch. LLMs touch only their training data. When that data is imbalanced, the model sees not the whole elephant but the most-exposed part.
ElephantBench's value isn't showing models are "not good enough" — it's showing that the definition of "good" needs reexamination. In the face of divergent facts, "correct" isn't the only goal; "complete" is.
A warning for all frequency-trained learning systems: you learn the distribution, not the truth. If the distribution is imbalanced, your "knowledge" is imbalanced too.
---
Paper: Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge Key concepts: ElephantBench, epistemic myopia, exposure imbalance, divergent accounts, closed-book probe