Blind Men and the Elephant: LLMs Show Epistemic Myopia on Long-Tail Divergent Knowledge

An Old Fable, a New Version

Blind men and an elephant: one touches a leg and says the elephant is like a pillar; one touches the trunk and says it's like a snake; one touches an ear and says it's like a fan. Each is partly right—and each misses the rest.

In August 2026, a paper brought this fable to LLMs: *Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge*.

Core finding: even the strongest frontier models, when facing multiple divergent accounts of the same fact, can recall both accounts simultaneously only 52.4% of the time. For nearly half the questions, the model remembers only the mainstream account and completely forgets the minority account.

LLMs don't know what they're missing—they know the mainstream version, but not what slipped away.

ElephantBench: 1,094 Divergent Questions

The paper built ElephantBench, containing 1,094 questions where different sources give different accounts of the same fact.

Example:

• Source A (IMDb): "Mother Teresa was born on August 26, 1910... born Agnes Gonxha Bojaxhiu."
• Source B (Poem of Quotes): "Mother Teresa, Agnes Gonxha Bojaxhiu, was born on August 27, 1910 in Skopje, Macedonia."

Both sources are human-verified and credible, yet they give different dates (Aug 26 vs Aug 27).

Evaluation format: closed-book. The model cannot see the source texts and must recall from parametric memory. The test is not "can the model answer" but "can it recall both divergent accounts."

This differs fundamentally from traditional QA, which assumes one correct answer per question. ElephantBench assumes multiple valid answers—the key is not "answering correctly" but "answering completely."

Construction: From Long-Tail Corpora to Divergent Questions

The pipeline, called "Tracing the Elephant":

1. Mine long-tail corpora: retrieve documents from low-exposure web corpora \(D_{low}\), typically outside mainstream training data. 2. Build a document graph: identify document pairs describing the same fact with different accounts; edges represent divergence. 3. Generate questions: for each divergent pair, generate a closed-book question whose answer exists in both sources but differs. 4. Human review: annotators verify that divergences are real and questions unambiguous. 5. Closed-book evaluation: models recall from parametric memory only.

The clever part: it reframes "long-tail" from "rare" to "divergent." Traditional long-tail benchmarks test whether models know rare facts; ElephantBench tests whether models can hold multiple versions of the same fact.

Epistemic Myopia Across 32 Models

Key numbers:

• Best model: recalls both accounts 52.4% of the time
• Average model: ≈ 39.4%
• Only the majority account recalled: ≈ 23.2% of model-question pairs
• Both accounts forgotten: ≈ 2.19%

Observations:

• Scale helps but doesn't fix it: full recall rises from 7B to 70B to frontier models, but plateaus around 50–60%.
• Chain-of-thought helps but doesn't fix it: even with extra "thinking time," models still drop minority accounts.
• Open vs closed weights: closed models (GPT, Claude) are more complete than open ones (Llama, Qwen), but the gap is mainly on minority accounts—open models match on majority-account recall.

Exposure Imbalance: The Root Cause

The deepest finding is the exposure imbalance effect. In training data, accounts of the same fact appear at different frequencies—"Aug 26" may appear 1,000 times while "Aug 27" appears 10 times. This maps directly onto recall:

• Higher majority-account exposure → model recalls only the majority
• Higher minority-account exposure → model recalls both

This explains why scale and reasoning can't fully cure myopia: the problem isn't model capability, it's the structure of training data. If the data is imbalanced, the model learns an imbalanced distribution. It's an information-theoretic version of garbage in, garbage out: the model isn't bad—the data is incomplete.

An Analogy: Survivorship Bias in Memory

In WWII, analysts plotted bullet holes on returning bombers—most on the wings. The intuition: armor the wings. But Abraham Wald pointed out the planes hit in the tail didn't return. You only see the survivors.

LLM parametric memory works the same way. The mainstream account is the "survivor" on the high-frequency side of training data; minority accounts are the non-survivors—present but drowned out. The model isn't unaware that minority accounts exist; the mainstream account's activation strength overwhelms them.

Connections to Existing Concepts

• Evaluation blind-spot law: traditional QA can't detect whether models remember divergent accounts—ElephantBench exposes a new "completeness blind spot."
• Judgment–gate decoupling: the model may internally encode both accounts (judgment) but only emit the mainstream one (gate). The fix is preserving diversity at output time—"fixing the gate is easier than fixing the judgment."
• Scalar illusion: accuracy is a scalar, but knowledge is a vector—which versions exist, their strengths, their relations. Managing a vector with a scalar is like taking blood pressure with a thermometer.
• New concept — epistemic myopia: a systematic, data-structure-driven bias, applicable to any frequency-trained system, including search engines and recommenders.

Practical Implications

1. Closed-book limits: real applications are often open-book (RAG). If retrieval results are also imbalanced, myopia likely persists. 2. Completeness beats correctness: for contested facts, "correct" is ill-defined—models should present both accounts. The alignment target becomes "output the full distribution," a harder goal. 3. Long-tail means divergent, not just rare—closer to real-world complexity, where many facts have multiple reasonable versions. 4. PPL as a cheap proxy: perplexity on divergent documents can approximate full-recall rate without running all 1,094 questions.

An Honest Assessment

Limitations:

• Scale: 1,094 questions vs MMLU's 14,000
• Long-tail definition depends on \(D_{low}\) corpus choice
• Closed-book ecological validity is limited
• 32 evaluated models may not include the newest frontier models

Still, as a conceptual contribution it does one important thing right: it makes "does the model remember divergent facts" an independent evaluation dimension, rather than burying it inside accuracy.

Conclusion

The fable's lesson: everyone sees only the part they touch. LLMs touch training data—and when that data is imbalanced, the model sees not the whole elephant but the most-exposed part.

ElephantBench's value isn't saying models are "not good enough"—it's saying the definition of "good" needs revisiting. For divergent facts, completeness matters as much as correctness.

A warning for all frequency-trained systems: you learn the distribution, not the truth. If the distribution is imbalanced, your "knowledge" is imbalanced.

---

Paper: Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge Key concepts: ElephantBench, epistemic myopia, exposure imbalance, divergent accounts, closed-book probe