Hallucinations Undermine Trust; Metacognition Is a Way Forward

Authors: Gal Yona¹, Mor Geva², Yossi Matias¹ — ¹Google Research, ²Tel Aviv University

arXiv:2605.01428v1 [cs.CL] 2 May 2026

1. Introduction

Despite significant progress in factual reliability, errors — commonly called "hallucinations" — remain a primary concern for generative AI, especially as LLMs are expected to help in increasingly complex or nuanced scenarios. Yet even in the simplest setting — factual QA with a clear ground truth — frontier models without external tools still hallucinate.

Core thesis: Most factuality gains in this domain come from expanding the model's knowledge boundary (encoding more facts) rather than improving awareness of that boundary (distinguishing known from unknown). The authors conjecture the latter is intrinsically hard: models may lack the discriminative ability to perfectly separate truth from error, creating an unavoidable trade-off between eliminating hallucinations and preserving utility.

2. Core Concept: Faithful Uncertainty

2.1 Redefining hallucination

• Traditional view: hallucination = any error
• New framing: hallucination = confident error (misinformation delivered without appropriate qualification)

This reveals a third path beyond the "answer or abstain" binary: express uncertainty.

2.2 What is faithful uncertainty?

Definition: Aligning linguistic uncertainty (what the model says) with intrinsic uncertainty (what the model "believes").

• Intrinsic uncertainty: the model's statistical confidence in the semantic content of its assertion
• Linguistic uncertainty: the confidence verbally expressed in the generated response

Example comparison:

| Type | Example | |------|---------| | Strict factuality | "Isa Genzken's Der Spiegel contains 121 photographs" (possibly wrong) | | Abstention | "I cannot confirm the number of photographs..." (utility loss) | | Faithful uncertainty | "I'm not entirely sure, but I believe Isa Genzken's Der Spiegel contains 121 photographs" (preserves utility, mitigates harm) |

3. Why Do Hallucinations Persist?

3.1 Theoretical upper bounds

Prior work argues extrinsic hallucinations are a structural inevitability of aut text generation:

• Halting-problem argument: No computable model can universally verify the truth of all facts (Banerjee et al., 2025; Xu et al., 2024)
• Calibrated models must hallucinate: Kalai & Vempala (2024) prove that calibrated models inevitably hallucinate when generating facts whose truth cannot be inferred from other facts
• Consistency–diversity trade-off: Kalavasis et al. (2025) show that pushing hallucination rates below a critical threshold requires drastically reducing output diversity, forcing mode collapse

3.2 The discriminative gap

Core question: knowing the average error rate (calibration) vs. knowing which specific instances are wrong (discrimination).

Empirical data:

• AUROC (ability to distinguish correct from incorrect answers) sits in the 0.70–0.85 range (Farquhar et al., 2024; Kang et al., 2025; Savage et al., 2025)
• Even at AUROC = 0.85, reducing the error rate from 25% to 5% requires discarding ~28% of valid answers
• The utility tax only becomes negligible at AUROC ≥ 0.95, which no current method achieves

3.3 Supporting anomalies

1. Poor generalization of truthfulness probes: model internals often lack the information needed to separate correct from incorrect answers (Levinstein & Herrmann, 2023; Marks & Tegmark; Orgad et al., 2025; Sky et al., 2024) 2. Confident hallucinations exist: factual errors with high intrinsic confidence (Simhi et al., 2025; Taubenfeld et al., 2025; Wang et al., 2025b) 3. Failure of advanced supervision: training models to "admit" mistakes does not transfer to hallucinations (Joglekar et al., 2025) 4. Reasoning increases hallucination: extended reasoning tends to increase hallucination rates and reduce abstention ability (Jaech et al., 2024; Li & Ng, 2025; Yao et al., 2025)

3.4 The empirical trade-off

Results on the SimpleQA Verified dataset show:

• Most frontier models lie along the diagonal, optimizing coverage at high hallucination rates
• Models attempting to maximize factuality are forced to sacrifice coverage, paying a utility tax
• The ideal "perfect discrimination" region (top-right) is entirely unoccupied

4. Metacognition

4.1 Definition

Metacognition: the ability to be aware of one's own uncertainty and act accordingly.

• Direct interaction: acting means honestly communicating uncertainty
• Agentic systems: acting means using uncertainty to guide when to search and what to trust

4.2 Why metacognition is critical for agents

Tool use seems to solve the knowledge problem: instead of encoding every fact, look it up. But:

• Without uncertainty awareness: the model cannot determine when to invoke tools (leading to inefficient overuse or dangerous underuse)
• Conflicting information cannot be weighed: when retrieval returns conflicting or low-quality information, the model cannot weigh it appropriately

Tools solve the storage problem but introduce a control problem: governing retrieval, verification, and orchestration.

4.3 The analogy with human metacognition

• Introspection: assessing one's own uncertainty
• Regulation: adjusting behavior based on that assessment

Contemporary agents often rely on static heuristics or over-engineered frameworks, whereas future agents in open environments need dynamic control.

5. Research Agenda

5.1 Challenges for metacognitive LLMs

1. Bootstrapping paradox: base models rarely express doubt naturally; SFT datasets are static, but "correct" uncertainty labels are dynamic relative to the model's current state 2. Preservation of signals: standard alignment techniques tend to induce mode-seeking behavior, making aligned models more overconfident than base models 3. Confidence attribution: a single scalar confidence score is insufficient; sources of uncertainty (aleatoric, epistemic, normative) must be distinguished 4. Causal evaluation: models may learn to mimic the style of faithful uncertainty rather than its substance 5. Agentic evaluation: evaluation must shift from end-to-end correctness to process-based control

5.2 Better evaluation of hallucination mitigation

1. Visualize utility–error trade-off curves: expose the utility sacrificed to achieve a given target error rate 2. Show frontier improvements: reject single-point comparisons (e.g., "we reached 95% accuracy") and demonstrate that a method yields more utility than baselines at a fixed error rate 3. Measure holistic spillover effects: test the "collateral damage" of interventions on general capabilities

6. Responses to Alternative Views

6.1 "We shouldn't deprioritize factuality"

Faithful uncertainty is not a substitute for knowledge expansion but a complement. The two are synergistic: a more knowledgeable model with good metacognition strictly dominates either capability alone.

6.2 "Users prefer confidence over uncertainty"

Faithful uncertainty does not target creative domains (where hallucination is genuinely desirable). In long-form generation, it need not be pervasive — it can be expressed locally, e.g., by flagging specific code lines or specific dates.

6.3 "Latent truth exists; we just need better probes"

Searching for latent truth is a worthwhile pursuit, but requires a strong assumption: a universal representation of truth over the entire long tail of facts. Given the evidence in §3.2, the authors are skeptical. Faithful uncertainty offers concrete room for improvement.

7. Discussion

• Fully eliminating hallucinations faces fundamental challenges due to the discriminative gap
• Faithful uncertainty serves as a complementary goal: models should be as knowledgeable as possible while faithfully expressing any remaining uncertainty
• Metacognitive awareness becomes increasingly important as LLMs evolve into agentic systems, serving as a control layer for robust tool use
• Connection to AI safety: at its core is honesty — requiring models to accurately represent their epistemic state rather than project false confidence
• A dual shift in model development and user expectations is needed

References (selected)

• Banerjee et al., 2025. LLMs will always hallucinate, and we need to live with this.
• Kalai & Vempala, 2024. Calibrated language models must hallucinate.
• Kalavasis et al., 2025. On the limits of language generation: Trade-offs between hallucination and mode-collapse.
• Farquhar et al., 2024. Detecting hallucinations in large language models using semantic entropy. *Nature*, 630(8017):625–630.
• Levinstein & Herrmann, 2023. Still no lie detector for language models.
• Simhi et al., 2025. HACK: Hallucinations along certainty and knowledge axes.
• Joglekar et al., 2025. Training LLMs for honesty via confessions.
• Haas et al., 2025. SimpleQA verified: A reliable factuality benchmark.
• Yona et al., 2024. Can LLMs faithfully express their intrinsic uncertainty in words?

*(Full reference list available in the original post.)*