Larger LLMs Confidently Pick Wrong Answers Even When They Know the Right One

Forum topic · 小凯 · 2026-05-22

Summary

A Seoul National University team (Yeom et al., arXiv:2605.22007) introduces a 'semantic probability mass' (P_mass) probe that aggregates every surface form of the correct answer in the model's pre-output distribution. Across 18 models (Qwen 0.8B–72B, Llama-3.1 8B–70B, plus base variants) on TriviaQA, NQ-Open, MMLU and ARC-Challenge, the authors find that 16%–47% of hallucinations in Instruct models are 'commitment failures': P_mass is at least 0.20, yet the model chooses something else. The failure rate rises monotonically with scale. Base models show no such trend, so the culprit is instruction tuning's 'sharpening' of the answer distribution. Failures mostly take the form of multi-token drift (Adam Smith → Adam Levine) rather than first-token misfires. A logistic probe on the t=1 hidden state reaches AUROC 0.61–0.87 for Instruct models, showing the correctness signal is pre-loaded. The work reframes hallucination as a structural commitment problem, not a knowledge gap.

Paper information

| Item | Detail | |---|---| | Title | Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer | | Authors | Jewon Yeom, Jaewon Sok, Heejun Kim, Seonghyeon Park, Jeongjae Park, Taesup Kim | | Affiliations | Seoul National University (Data Science, Rural Systems Engineering, Aerospace Engineering); GIST (Computer Science) | | arXiv | 2605.22007 | | Submitted | 21 May 2026 | | Core claim | 16%–47% of hallucinations in Instruct LLMs are commitment failures — the correct answer carries sufficient probability mass at the output moment, yet the model does not select it. This share rises monotonically with scale. The driver is not raw size but instruction tuning's sharpening of answer commitments. |

Key points

1. Probing the second before the first token

The standard story is that an LLM hallucinates because the correct token simply isn't in its distribution — a knowledge gap to be filled. Simhi et al. (2025, "CHOKE") and Farquhar et al. (2024, semantic entropy) already noted that "confidently wrong" is a distinct phenomenon from "unsure and confabulating."

This paper intercepts the distribution at the instant before the first answer token is emitted. The authors define semantic probability mass ( $P_{\text{mass}}$ ) as the sum of probabilities of all surface variants of the correct answer — full names, abbreviations, aliases, different capitalizations — pooled across the vocabulary.

Worked example: the answer is *Saint Petersburg*. The model assigns 24% to "Saint", 11% to "St", 13% to "C" (start of *St. Basil's Cathedral*). A competitor "Mos" sits at 31%, so greedy decoding outputs *Moscow*. Yet $P_{\text{mass}} = 0.24 + 0.11 + 0.13 = 0.48$ . The correct mass sits plainly in the distribution.

A commitment failure is defined as $P_{\text{mass}} \geq 0.2$ with a wrong final answer.

2. Scale curve: 16% → 47%

Qwen3.5-0.8B Instruct: 8.3% accuracy, 16% of hallucinations are commitment failures
Qwen3.5-9B Instruct: 29.4% accuracy, 32%
Qwen2.5-72B Instruct: 36.4% accuracy, 41%
Llama-3.1-8B Instruct: 32.8% accuracy, 33%
Llama-3.1-70B Instruct: 44.7% accuracy, 47%

Commitments break in two patterns:

First-token misfire (~20% of failures, rising with scale): the very first token is already off-track, e.g. *Mos* instead of any *Saint*
Multi-token drift (~80%): the first token lands on a valid alias but subsequent tokens walk off the right answer (Adam Smith → Adam Levine; George Washington → George Washington Carver)

3. Size isn't the culprit — instruction tuning is

Instruct models: rises monotonically from 0.31 (0.8B) to 0.57 (72B) for both Qwen and Llama families
Base models: flat curve, 0.26–0.33, independent of size

Instruction tuning exerts a "sharpening" pressure on the answer distribution: instead of letting probability diffuse across aliases, the model concentrates it onto the few tokens that look "most right." This is a double-edged sharpening — equally good for decisive correctness and decisive wrongness. The authors phrase it bluntly: instruction tuning makes helpfulness and confident hallucination two consequences of the same underlying disposition.

4. Why some 20%-mass samples succeed and others fail

P_{\text{mass}}

Correct-output group: the highest-probability alias averages 0.78. The mass collapses cleanly onto one surface form and beats all competitors.
Commitment-failure group: the highest-probability alias averages only 0.26. Aliases are fragmented — *Saint*, *St*, *C* — and none individually outweighs the distractor's single token.

The effect size (Cohen's $$d$$ ) is negative for all 18 models without exception. For Instruct models, $$|d|$$ grows from 1.34 to 4.30 with scale. Small-model failure is fragmentation-driven; large-model failure is sharpened-distractor-driven. Same mechanism, different stop on the failure surface.

5. The model already knows — in its hidden state

before

AUROC 0.61–0.87

Attention patterns corroborate this: Instruct models at $$t=1$$ direct a larger share of attention (+0.09) toward question tokens, behaving like a "dictionary lookup." Base models first emit filler ("Answer:", "The") and only later commit to searching for the answer.

6. Implications and honest limits

P_{\text{mass}}

What the framework buys the field:

Adds a structural dimension to hallucination research: not just *does the model know?* but *which surface forms does it commit to?*
Reframes at least part of hallucination as a decision error over a known distribution rather than a knowledge gap
Warns alignment work that helpful decisiveness and confident hallucination share a common sharpening mechanism; suppressing one without the other will require concept-aware correction at the sharpening step

Two research directions the paper sketches:

Concept-aware decoding: lift greedy argmax from a single token to a semantically equivalent token family
Layered commitment detection: identify commitment points beyond the answer itself (domain, entity, rhetorical frame), each opening its own intervention window

References

1. Yeom, J. et al. (2026). *Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer.* arXiv:2605.22007. 2. Simhi, A. et al. (2025). *Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer.* EMNLP 2025 Findings. 3. Farquhar, S. et al. (2024). *Detecting Hallucinations in Large Language Models Using Semantic Entropy.* Nature, 630, 625–630. 4. Calderon, N. et al. (2026). *Empty Shelves or Lost Keys? Recall is the Bottleneck for Parametric Factuality.* arXiv:2602.14080. 5. Ouyang, L. et al. (2022). *Training Language Models to Follow Instructions with Human Feedback.* NeurIPS 2022.