Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer

> Imagine showing a student their scratch paper—the correct answer is clearly written there, just slightly messy, next to a pile of synonyms and abbreviations. They hesitate for two seconds, then pick a more "prominent" wrong option. When asked "didn't you know it?", they can't answer. A recent paper does something similar to 18 LLMs.

Paper Information

| Item | Content | |------|---------| | Title | Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer | | Authors | Jewon Yeom, Jaewon Sok, Heejun Kim, Seonghyeon Park, Jeongjae Park, Taesup Kim | | Institutions | Seoul National University (Graduate School of Data Science, Dept. of Biosystems Engineering, Dept. of Aerospace Engineering), GIST (School of Electrical Engineering and Computer Science) | | arXiv | 2605.22007 | | Core claim | 16%–47% of Instruct LLM hallucinations are "commitment failures"—at the moment of output, the probability distribution contains substantial probability mass on the correct answer, but the model doesn't choose it. This share rises monotonically with model size. The driver is not scale itself but the "sharpening" effect of instruction tuning on answer commitment. |

1. Dissecting hallucination: look at the moment before output

The standard view holds that LLMs hallucinate because they lack knowledge—the correct token simply isn't in their probability distribution. But prior work complicates this: Simhi et al. (2025) documented "CHOKE" behavior where models confidently output wrong answers despite knowing the correct one, and Farquhar et al.'s semantic entropy work distinguished "confidently wrong" from "uncertainly fabricated."

This paper intercepts the model's probability distribution at the moment it is about to emit the first answer token. The authors define semantic probability mass (\(P_{\text{mass}}\)): the summed token probability over all surface forms of one answer—full names, abbreviations, aliases, casing variants.

Example: for Saint Petersburg, the model may assign 24% to "Saint", 11% to "St", and 13% to "C"—each individually below a rival's "Mos" (31%), so greedy decoding outputs Moscow. But summed: 0.24 + 0.11 + 0.13 = 0.48. The correct answer's mass was there all along.

This is called commitment failure: \(P_{\text{mass}} \geq 0.2\) yet the final output is wrong.

2. From 16% to 47%: larger models "know but don't choose"

Across Instruct versions of Qwen and Llama families, commitment failures as a share of all hallucinations climb from 16% (0.8B) to 47% (70B):

• Qwen3.5-0.8B Instruct: 8.3% accuracy; 16% of hallucinations are commitment failures
• Qwen3.5-9B Instruct: 29.4% accuracy; 32%
• Qwen2.5-72B Instruct: 36.4% accuracy; 41%
• Llama-3.1-8B Instruct: 32.8% accuracy; 33%
• Llama-3.1-70B Instruct: 44.7% accuracy; 47%

Small models err mostly because they genuinely don't know. Larger models, nearly half the time, err despite knowing.

Commitment failures split into two token-level types:

1. First-token selection failure (~20%): the very first token lands on an unrelated word (answer is Saint Petersburg, model emits "Mos" toward Moscow). This share grows with scale. 2. Multi-token divergence (~80%): the first token follows a valid alias, but later tokens derail (Adam Smith → "Adam Levine"; George Washington the president → "George Washington Carver").

3. It's not scale—it's instruction tuning

Base models tell a different story. For first-token selection failures:

• Instruct models: the erroneous token's probability rises monotonically from 0.31 (0.8B) to 0.57 (72B), in both Qwen and Llama families.
• Base models: flat, fluctuating between 0.26 and 0.33 regardless of size.

Scale itself doesn't make models wrongly confident; instruction tuning does. The paper describes instruction tuning as sharpening answer commitment—concentrating the distribution onto a few "most plausible-looking" tokens. When sharpening lands on the correct answer, the model is decisive and accurate; when it lands on a wrong one, the model is equally decisive—just decisively wrong. Helpfulness and confident hallucination are two consequences of the same underlying disposition.

4. The structure of the mass matters, not its size

Among samples with similar \(P_{\text{mass}}\), why do some succeed and others fail? The shape of the distribution:

• Correct samples: the highest-probability alias token averages 0.78—mass is concentrated on one form.
• Failure samples: the top alias token averages only 0.26—mass is scattered (Saint / St / C each hold a share), and no single token beats the competitor.

Cohen's \(d\) for this difference is negative in all 18 models—zero exceptions—and \(|d|\) grows from 1.34 to 4.30 with scale in Instruct models. Small-model failures stem from fragmentation; large-model failures occur even when the correct answer has collapsed onto one token, because the wrong concept's token is sharpened even harder.

5. The model "knows" before it speaks—in its hidden states

For Instruct models, a simple logistic regression probe on the last-layer hidden state at \(t=1\)—before the first token is generated—predicts whether the final answer is correct, with MCQA AUROC of 0.61–0.87. Base models achieve only 0.50–0.63.

The correctness signal is front-loaded into hidden representations. Since the hidden-level gap (+0.08) is much smaller than the output-level gap (+0.29), much of instruction tuning's effect lies in the projection from representation to tokens. Attention patterns agree: Instruct models attend more to question tokens at \(t=1\) (+0.09), as if "looking up a dictionary," while Base models emit filler ("Answer:", "The") before searching.

6. What it means—and what it doesn't claim

The paper honestly notes that \(P_{\text{mass}}\) is an analytical probe, not a deployable detector—it requires the set of correct-answer aliases as input, which isn't available before the model answers.

But its methodological implications are significant:

• It reframes hallucination research: instead of asking "does the model know the answer?", we can ask "which versions of the answer does the model hold, and where does it concentrate the mass?" Hallucination is partly a decision-level failure in the probability distribution, not merely missing knowledge.
• It is a warning for alignment: instruction tuning makes models decisive. Decisively right is good; decisively wrong is not. Helpfulness and confident hallucination are two faces of the same coin.
• Proposed directions include concept-aware decoding (lifting greedy argmax from single tokens to semantically equivalent token families) and finer-grained commitment-moment identification (domain commitment, entity commitment, rhetorical framing commitment), each offering different intervention windows.

References

1. Yeom, J. et al. (2026). Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer. arXiv:2605.22007. 2. Simhi, A. et al. (2025). Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer. EMNLP 2025 Findings. 3. Farquhar, S. et al. (2024). Detecting Hallucinations in Large Language Models Using Semantic Entropy. *Nature*, 630, 625–630. 4. Calderon, N. et al. (2026). Empty Shelves or Lost Keys? Recall is the Bottleneck for Parametric Factuality. arXiv:2602.14080. 5. Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.