Hallucinations Undermine Trust; Metacognition is a Way Forward

Authors: Gal Yona¹, Mor Geva², Yossi Matias¹ (¹ Google Research, ² Tel Aviv University)

arXiv: 2605.01428v1 [cs.CL] 2 May 2026

1. Introduction

Despite significant progress in factual reliability, errors—commonly called "hallucinations"—remain a major concern for generative AI, even in the simplest settings such as factoid question answering with clear ground truth, where frontier models without external tools still hallucinate.

Core thesis: Most factual gains in this domain have come from expanding models' knowledge boundaries (encoding more facts), rather than improving awareness of those boundaries (distinguishing known from unknown). The authors conjecture the latter is inherently hard: models may lack the discriminative capacity to perfectly separate truth from error, creating an unavoidable trade-off between eliminating hallucinations and preserving utility.

2. Core Concept: Faithful Uncertainty

2.1 Redefining Hallucination

• Traditional view: hallucination = any error
• New framework: hallucination = confident error (misinformation without appropriate hedging)

This reveals a third path beyond the "answer or abstain" binary: expressing uncertainty.

2.2 What is Faithful Uncertainty?

Definition: Aligning linguistic uncertainty (what the model says) with intrinsic uncertainty (what the model "believes").

• Intrinsic uncertainty: the model's statistical confidence in the semantic content of its assertion
• Linguistic uncertainty: the degree of confidence the model verbally expresses in its generated response

Comparison of approaches:

| Type | Example | |------|---------| | Strict factuality | "Isa Genzken's Der Spiegel contains 121 photos" (possibly wrong) | | Abstention | "I cannot confirm the number of photos..." (loses utility) | | Faithful uncertainty | "I'm not entirely sure, but I believe Isa Genzken's Der Spiegel contains 121 photos" (preserves utility, mitigates harm) |

3. Why Do Hallucinations Persist?

3.1 Theoretical Upper Bounds

Prior work argues extrinsic hallucinations are a structural inevitability of autoregressive text generation:

• Halting problem argument: No computable model can universally verify the truth of all facts (Banerjee et al., 2025; Xu et al., 2024)
• Calibrated models must hallucinate: Kalai & Vempala (2024) prove that calibrated models necessarily hallucinate when generating facts whose truth cannot be inferred from other facts
• Consistency-diversity trade-off: Kalavasis et al. (2025) show that pushing the hallucination rate below a critical threshold requires drastically reducing output diversity, forcing mode collapse

3.2 The Discriminative Gap

Core question: knowing the average error rate (calibration) vs. knowing which specific instances are wrong (discrimination).

Empirical data:

• AUROC (ability to distinguish correct from incorrect answers) sits in the 0.70–0.85 range (Farquhar et al., 2024; Kang et al., 2025; Savage et al., 2025)
• Even at AUROC = 0.85, reducing error from 25% to 5% requires discarding roughly 28% of valid answers
• Only at AUROC ≥ 0.95 does the utility tax become negligible—and no current method reaches this level

3.3 Supporting Anomalies

1. Poor generalization of truthfulness probes: models' internal states often lack the information needed to distinguish correct from incorrect answers (Levinstein & Herrmann, 2023; Marks & Tegmark; Orgad et al., 2025; Sky et al., 2024) 2. Confident hallucinations exist: factual errors with high intrinsic confidence (Simhi et al., 2025; Taubenfeld et al., 2025; Wang et al., 2025b) 3. Failure of advanced supervision: training models to "admit" errors does not transfer to hallucinations (Joglekar et al., 2025) 4. Reasoning increases hallucination: extended reasoning tends to increase hallucination rates and reduce abstention (Jaech et al., 2024; Li & Ng, 2025; Yao et al., 2025)

3.4 The Empirical Trade-off

On the SimpleQA Verified dataset:

• Most frontier models distribute along the diagonal, optimizing coverage at high hallucination rates
• Models attempting to maximize factuality are forced toward abstention, paying a utility tax
• The ideal "perfect discrimination" region is entirely unoccupied

4. Metacognition

4.1 Definition

Metacognition: the ability to be aware of one's own uncertainty and act on it.

• Direct interaction: acting means honestly communicating uncertainty
• Agentic systems: acting means using uncertainty to guide when to search and what to trust

4.2 Why Metacognition is Critical for Agents

Tool use seems to solve the knowledge problem—instead of encoding every fact, the model can look it up. But:

• Without uncertainty awareness: the model cannot determine when to invoke tools (leading to inefficient overuse or dangerous underuse)
• Conflicting information cannot be weighed: when retrieval returns conflicting or low-quality information, the model cannot weigh it appropriately

Tools solve the storage problem but introduce a control problem: governing retrieval, verification, and orchestration.

4.3 Analogy with Human Metacognition

• Introspection: assessing one's own uncertainty
• Regulation: adjusting behavior based on that assessment

Contemporary agents often rely on static heuristics or over-engineered frameworks, while future agents in open environments need dynamic control.

5. Research Recommendations

5.1 Challenges for Metacognitive LLMs

1. Bootstrapping paradox: base models rarely express skepticism naturally; SFT datasets are static, but "correct" uncertainty labels are dynamic relative to the model's current state 2. Preserved signals: standard alignment techniques tend to induce mode-seeking behavior, making aligned models more overconfident than base models 3. Confidence attribution: a single scalar confidence score is insufficient; sources of uncertainty (aleatoric, epistemic, normative) must be distinguished 4. Causal evaluation: models may learn to mimic the style rather than the substance of faithful uncertainty 5. Agentic evaluation: evaluation must shift from end-to-end correctness to process-based control assessment

5.2 Better Evaluation of Hallucination Mitigation

1. Visualize utility–error trade-off curves: expose the utility sacrificed to achieve a target error rate 2. Show frontier improvements: reject single-point comparisons (e.g., "we achieved 95% accuracy") in favor of demonstrating higher utility than baselines at a fixed error rate 3. Measure overall spillover effects: test the "collateral damage" of interventions on general capabilities

6. Responding to Alternative Views

6.1 "We shouldn't deprioritize factuality"

Response: Faithful uncertainty is not a substitute for knowledge expansion but a complement. The two are synergistic: a more knowledgeable model with good metacognition strictly outperforms either capability alone.

6.2 "Users prefer confidence over uncertainty"

Response: Faithful uncertainty does not target creative domains (where hallucination is desirable). In long-form generation, it need not be intrusive—uncertainty can be expressed locally by flagging specific code lines or specific dates.

6.3 "Latent truth exists; we just need better probes"

Response: Searching for latent truth is a worthwhile pursuit, but it requires strong assumptions: that a universal representation of truth exists across the long tail of facts. Given the evidence in §3.2, the authors are skeptical. Faithful uncertainty offers concrete room for improvement.

7. Discussion

• Fully eliminating hallucinations faces fundamental challenges due to the discriminative gap
• Faithful uncertainty serves as a complementary goal: models should be as knowledgeable as possible while faithfully expressing any remaining uncertainty
• Metacognitive awareness becomes increasingly important as LLMs evolve into agentic systems, serving as a control layer for robust tool use
• Connection to AI safety goals: at its core, this is about honesty—requiring models to accurately represent their epistemic state rather than project false confidence
• A dual shift in model development and user expectations is needed

References (Selected)

• Banerjee et al., 2025. LLMs will always hallucinate, and we need to live with this.
• Kalai & Vempala, 2024. Calibrated language models must hallucinate.
• Kalavasis et al., 2025. On the limits of language generation: Trade-offs between hallucination and mode-collapse.
• Farquhar et al., 2024. Detecting hallucinations in large language models using semantic entropy. *Nature*, 630(8017):625–630.
• Simhi et al., 2025. HACK: Hallucinations along certainty and knowledge axes.
• Joglekar et al., 2025. Training LLMs for honesty via confessions.
• Haas et al., 2025. SimpleQA verified: A reliable factuality benchmark.
• Yona et al., 2024. Can LLMs faithfully express their intrinsic uncertainty in words?

*(Full reference list available in the original post.)*