Summary
A new paper (arXiv:2605.05166) by Mina Gabriel shows that first-token confidence (phi_first) — the normalized entropy of the top-K logits at the first content-bearing answer token of a single greedy decode — can detect hallucinations as well as or better than multi-sample methods like semantic self-consistency, which require repeated decoding and external natural language inference overhead. Across three 7-8B instruction-tuned models and two benchmarks on closed-book short-answer factual QA, phi_first achieves a mean AUROC of 0.820, versus 0.793 for semantic agreement and 0.791 for standard surface-form self-consistency. A subsumption test shows phi_first correlates moderately to strongly with semantic agreement, and combining the two signals yields only marginal gains, suggesting most uncertainty information from multi-sample agreement is already present in the initial token distribution. The author argues phi_first should be a default low-cost baseline before sampling-based uncertainty estimation.
Paper Overview
- Field: NLP
- Author: Mina Gabriel
- Published: 2026-05-06
- arXiv: 2605.05166
Abstract
Self-consistency detects hallucinations by generating multiple sampled answers to a question and measuring agreement, but this requires repeated decoding and can be sensitive to lexical variation. Semantic self-consistency improves this by clustering sampled answers by meaning using natural language inference, but it adds both sampling cost and external inference overhead.The paper shows that first-token confidence, phi_first, computed from the normalized entropy of the top-K logits at the first content-bearing answer token of a single greedy decode, matches or modestly exceeds semantic self-consistency on closed-book short-answer factual question answering.
Key Results
- Across three 7-8B instruction-tuned models and two benchmarks, phi_first achieves a mean AUROC of 0.820, compared with 0.793 for semantic agreement and 0.791 for standard surface-form self-consistency.
- A subsumption test shows phi_first is moderately to strongly correlated with semantic agreement, and combining the two signals yields only a small AUROC improvement over phi_first alone.
- These results suggest that much of the uncertainty information captured by multi-sample agreement is already available in the model's initial token distribution.
Conclusion
The author argues that phi_first should be reported as a default low-cost baseline before invoking sampling-based uncertainty estimation.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619593