Uncertainty Is Not Hallucination: Systematic Evaluation Shows UE Signals Often Fail to Detect LLM Hallucinations

Forum topic · 小凯 · 2026-05-27

Summary

A widely held assumption in the LLM community is that model uncertainty correlates with hallucination: if a model is less confident, its output is more likely wrong. A new systematic study, "Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination" (arXiv:2605.27016), tests this assumption with 46 uncertainty estimators across 4 hallucination datasets (RAGTruth, PreciseWikiQA, LongWiki, NonExistentRefusal) and 3 open-source models (Mistral-7B-Instruct, Llama-2-7B/13B-Chat). Findings: (1) no estimator is universally best—rankings flip across tasks; (2) dataset (hallucination type), not model, is the primary source of ranking variance (Kendall's tau as low as 0.10 across datasets vs. up to 0.71 across models); (3) on intrinsic hallucinations (RAG context unfaithfulness), uncertainty signals barely beat random baselines, exposing a structural blind spot for RAG hallucination detection; (4) top estimators form three complementary signal clusters (probabilistic/sampling, NLI-graph, attention-based). The paper's practical message: validate any uncertainty estimator on your own task-specific data before deploying it as a hallucination detector, since cross-task transfer is likely to fail.

A long-standing intuition in the LLM community holds that when a model is "unsure," its output is more likely to be wrong—and that uncertainty estimation (UE) can therefore serve as a hallucination detector. A systematic study set out to verify this default assumption, and largely refuted it.

The paper

| Item | Detail | |------|--------| | Title | Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination | | Authors | Yedidia Agnimo, Anna Korba, Annabelle Blangero, Nicolas Chesneau, Karteek Alahari | | Affiliations | Ekimetrics, CREST/ENSAE (Institut Polytechnique de Paris), Inria (Université Grenoble Alpes) | | arXiv ID | 2605.27016 | | Category | cs.CL |

Key points

The assumption: higher model uncertainty → higher hallucination probability. If true, hallucination detection reduces to a scalar threshold on uncertainty scores. Dozens of UE methods were built over three years—information-theoretic, sampling-based, hidden-state, density-based, and self-reflection approaches—but their design goal was confidence quantification, not hallucination detection.
Experimental matrix: 46 estimators across six families (e.g., Maximum Sequence Probability, Perplexity, Token Entropy, Claim-Conditioned Probability, Semantic Entropy, SAR, AttentionScore, EigenScore, Mahalanobis Distance, PTrue, Eccentricity), evaluated on 4 datasets covering two hallucination definitions and 3 open models (Mistral-7B-Instruct, Llama-2-7B-Chat, Llama-2-13B-Chat).
Finding 1 — rankings flip across tasks: CCP and CocoaMSP offer the best stability/AUROC balance, but even top estimators collapse on some task–model panels; MSP can go from first place to outside the top 20.
Finding 2 — dataset matters more than model: switching datasets with a fixed model yields mean Kendall's tau of only 0.10–0.15 (near-uncorrelated); switching models on a fixed dataset yields much higher agreement (up to tau = 0.71 on LongWiki, 0.62 on PreciseWikiQA).
Finding 3 — intrinsic hallucinations are nearly invisible: on RAGTruth (context faithfulness), uncertainty-based detection barely exceeds the 0.5 random AUROC baseline, while on short-form QA (PreciseWikiQA) nearly all estimators perform well. UE captures "the model doesn't know what to say" but not "the model wrongly believes it has supporting evidence"—the typical RAG hallucination.
Three signal clusters: (1) probabilistic/sampling methods (MSP, CCP, CocoaMSP, SAR, rho ≥ 0.78); (2) NLI-graph methods (Eccentricity-c/e, rho = 0.36); (3) AttentionScore, uncorrelated with the others. Combining representatives from different clusters maximizes signal diversity.

Honest limitations

Only 7B–13B open models were tested; extrapolation to larger closed models is unsupported.
No unified theory explains why some estimator families succeed on short QA but fail elsewhere.
LLM-as-a-Judge (Llama-3.1-70B) ground truth for LongWiki and NonExistentRefusal leaves a partial evaluation loop unclosed.
Generation-length effects as a confound are not fully isolated.
Why attention-based signals are independent remains unexplained.

Takeaway

This is a "stop" paper: rather than proposing a new method, it systematically shows that methods the community already uses fail in most settings. The recommendation is not to abandon uncertainty estimation, but to validate any estimator on your own task-specific data before deploying it—cross-task transfer likely fails, and context-unfaithful but fluent generations remain a structural blind spot of all current uncertainty frameworks.

References 1. Agnimo et al., "Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination", arXiv:2605.27016, 2026. 2. Bang et al., "HalluLens: LLM Hallucination Benchmark", ACL 2025. 3. Niu et al., "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models", ACL 2024. 4. Kadavath et al., "Language Models (Mostly) Know What They Know", NeurIPS 2022. 5. Farquhar et al., "Detecting Hallucinations in Large Language Models Using Semantic Entropy", Nature, 2024.