Uncertainty About Uncertainty: LLM Confidence and Hallucinations Are Not the Same Thing

Forum topic · 小凯 · 2026-05-27

Summary

A systematic study on arXiv (2605.27016) by researchers from Ekimetrics, CREST/ENSAE, and Inria evaluates whether uncertainty estimation (UE) signals can reliably detect LLM hallucinations. The authors benchmark 46 uncertainty estimators across 4 hallucination datasets and 3 open-source models (Mistral-7B-Instruct, Llama-2-7B-Chat, Llama-2-13B-Chat), covering intrinsic (context-faithfulness) and extrinsic (knowledge-consistency) hallucinations. Key findings: the correlation between uncertainty and hallucination is highly variable and often weak; no single estimator performs best across all tasks; dataset/hallucination type—far more than model choice—drives estimator ranking (Kendall's tau drops to 0.10–0.15 across datasets, but reaches 0.62–0.71 across models on fixed datasets); and intrinsic hallucinations such as RAG context fabrication are a structural blind spot for UE methods. The paper groups top estimators into three complementary signal clusters and advises validating any estimator on task-specific data before deployment. This is a widely shared Chinese forum post analyzing and contextualizing the paper's methodology, results, and limitations.

Overview

A widely-circulated idea in the LLM world: if a model "lacks confidence" when generating text, its output is more likely to be wrong; if the model is very confident, it should be telling the truth. Based on this intuition, dozens of uncertainty estimation (UE) methods have been developed—checking logit probabilities, sampling variability, hidden-state geometry, even directly asking the model "are you sure?"

A new paper systematically tests this default assumption—and finds it mostly fails.

| Item | Detail | |---|---| | Paper | Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination | | Authors | Yedidia Agnimo, Anna Korba, Annabelle Blangero, Nicolas Chesneau, Karteek Alahari | | Affiliations | Ekimetrics, CREST/ENSAE (Institut Polytechnique de Paris), Inria (Université Grenoble Alpes) | | arXiv ID | arXiv:2605.27016 | | Category | cs.CL |

Experimental Design

46 uncertainty estimators spanning six families: information/logit-based (Maximum Sequence Probability, Perplexity, Token Entropy, Claim-Conditioned Probability), white-box sampling semantic dispersion (Semantic Entropy, Cocoa, SAR), internal-state (AttentionScore, EigenScore), density-based (Mahalanobis Distance), reflection (PTrue), and black-box NLI-graph methods (Eccentricity, EigValLaplacian).
3 models: Mistral-7B-Instruct, Llama-2-7B-Chat, Llama-2-13B-Chat.
4 datasets covering two hallucination definitions: RAGTruth (intrinsic — context faithfulness), PreciseWikiQA (extrinsic, short-form), LongWiki (extrinsic, long-form), NonExistentRefusal (refusal of unseen entities).

Key Findings

Estimator rankings flip across tasks. CCP and CocoaMSP offer the best AUROC/stability balance; MSP has higher average AUROC but large ranking variance, dropping out of the top 20 on some panels.
Dataset matters far more than model. Switching datasets with a fixed model yields Kendall's tau of only 0.10–0.15 (nearly uncorrelated); switching models on a fixed dataset yields tau up to 0.71 (LongWiki) and 0.62 (PreciseWikiQA). Estimator choices do not transfer across task types.
Intrinsic hallucinations are nearly invisible to UE. On RAGTruth, detection ability is weakest across all four datasets, with information-based methods barely above the 0.5 random AUROC baseline. On short-form QA (PreciseWikiQA), nearly all estimators far exceed the baseline. UE detects "the model is unsure what to say" but not "the model wrongly believes it has supporting evidence"—the typical RAG hallucination.
Three signal clusters. Top estimators group into (1) logit/sampling-based (MSP, CCP, CocoaMSP, SAR; inter-correlation ρ ≥ 0.78), (2) NLI-graph-based (Eccentricity-c and Eccentricity-e; ρ = 0.36), and (3) internal-state (AttentionScore, nearly uncorrelated with everything else). Choosing representatives from different clusters maximizes signal complementarity.

Honest Limitations

Only 7B–13B open-source models were tested; extrapolation to larger closed-source models is unsupported.
No unified theory explains *why* certain methods fail on certain tasks.
Ground truth for LongWiki and NonExistentRefusal relied on LLM-as-a-Judge (Llama-3.1-70B); the evaluation loop is not fully closed.
Generation length effects as a confounder were not fully isolated.
Why AttentionScore captures an independent signal remains unexplained.

Takeaway

The paper's message is not "abandon uncertainty detection" but: validate any uncertainty estimator on your own data before deploying it in a hallucination-detection pipeline. Cross-task transfer will likely fail, no estimator is universal, and some hallucinations—fluent fabrication without contextual support—are a structural blind spot of every existing uncertainty framework.

References

1. Agnimo, Korba, Blangero, Chesneau & Alahari, "Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination", arXiv:2605.27016, 2026. 2. Bang et al., "HalluLens: LLM Hallucination Benchmark", ACL 2025. 3. Niu et al., "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models", ACL 2024. 4. Kadavath et al., "Language Models (Mostly) Know What They Know", NeurIPS 2022. 5. Farquhar et al., "Detecting Hallucinations in Large Language Models Using Semantic Entropy", Nature, 2024.