The First Token Knows: Single-Decode Confidence Beats Semantic Entropy at 1/11 the Cost

The hallucination-detection industry has a standard ritual: sample the same question 10 times (temperature 0.7, top-p 0.95), feed the answers into a DeBERTa NLI model for semantic clustering, and treat dispersion across semantic clusters as a sign the model is guessing. This is Semantic Entropy — cited hundreds of times, spawning an entire family of methods (SEU, KLE, DSE).

Mina Gabriel ran an experiment that makes all of this look a bit awkward. She looked only at the probability distribution over the model's first content-bearing answer token.

No 10× sampling. No NLI clustering. No external model. No temperature tuning. Just one token.

AUROC 0.820. Semantic Entropy: 0.793. At 1/11 the cost.

The Results

| Method | Generation calls | Extra model | PopQA | TriviaQA | Overall AUROC | |--------|-----------------|-------------|-------|----------|-------------------| | Verbalized confidence | 1× | none | 0.705 | 0.695 | 0.700 | | Surface-form SC | 11× | none | 0.834 | 0.748 | 0.791 | | Semantic AU | 11× | DeBERTa NLI | 0.839 | 0.748 | 0.793 | | φ_first (this paper) | 1× | none | 0.875 | 0.764 | 0.820 |

Three 7–8B models (Llama-3.1-8B, Mistral-7B-v0.3, Qwen2.5-7B), two benchmarks (PopQA, TriviaQA), 1,000 questions each. φ_first ranks first in 5/6 model-dataset combinations — 2.7 AUROC points above Semantic Entropy and 12 points above verbalized confidence, with zero extra overhead.

The Core Insight: The First Commitment Moment

The field has assumed model uncertainty is "distributed" and only revealed through repeated sampling. Gabriel's claim: the model makes its earliest commitment at the first meaningful answer token.

Ask "Who wrote Hamlet?" — if the first token is "Shakespeare" at 95% probability, the model knows. If "Shakespeare" is 20%, "Marlowe" 18%, "The" 15%, the model is guessing — even if the rest of the answer reads fluently.

φ_first is the normalized entropy over top-K (K=100) probabilities at the first content-bearing token (skipping whitespace, punctuation, and template prefixes). Values range 0 (uniform — maximally uncertain) to 1 (fully concentrated — maximally confident), where:

The elegance: no sampling, no external model, no semantic parsing. Just how hesitant the model is when it writes its first word.

The Kill Shot: +0.02

φ_first and Semantic AU are moderately-to-strongly correlated (Pearson r = 0.54–0.76 across the six model-dataset combinations). Ensembling both via logistic regression improves AUROC by only +0.02.

That means Semantic Entropy's discriminative information is almost entirely captured by the first token. Multi-sampling plus NLI clustering is essentially an expensive recomputation of uncertainty already exposed by a single decode — theoretically correct, but over-engineered.

Is It Just an Answer-Length Proxy?

A natural objection: does φ_first just encode "short answer = confident"? Partial correlation analysis controlling for answer correctness:

| Dataset | Model | Raw r | Partial (correctness-controlled) | |---------|-------|-------|----------------------------------| | PopQA | Llama | -0.16 | -0.02 | | PopQA | Mistral | -0.13 | -0.03 | | PopQA | Qwen | -0.14 | -0.04 | | TriviaQA | Llama | -0.23 | -0.18 | | TriviaQA | Mistral | -0.25 | -0.17 | | TriviaQA | Qwen | -0.11 | -0.05 |

On PopQA, partial correlations collapse to near zero — φ_first genuinely measures internal uncertainty. Residual length sensitivity on TriviaQA is honestly listed as a limitation.

Uncomfortable Implications

1. The subfield may need an Occam's razor. If Semantic Entropy's lexical-vs-semantic distinction can be approximated by first-token entropy, the incremental value of SEU, KLE, DSE and successors needs rigorous re-examination. The ensemble gain of only +0.02 AUROC suggests that contribution is nearly zero in closed-domain short-answer QA. 2. The 11× cost gap means wasted compute. At GPT-4-class API prices, switching from Semantic AU to φ_first saves roughly 90% of generation cost per uncertainty estimate. 3. There is an early signal between confidence and correctness — and the field ignored it. The seed of a factual answer's truthfulness is planted in the very first word.

Statistical Rigor

Bootstrap testing (1,000 paired resamples): φ_first significantly beats Semantic AU in 3/6 cells (p<0.05) and ties in the rest — hence the paper's careful phrasing "matches or modestly exceeds." Honest science, and strong evidence.

Paper Details

| Item | Content | |------|---------| | Title | The First Token Knows: Single-Decode Confidence for Hallucination Detection | | Author | Mina Gabriel | | Affiliation | Temple University, Philadelphia | | arXiv | 2605.05166 | | Published | 2026-05-06 | | Method | φ_first: first-token top-K normalized entropy | | Benchmarks | PopQA / TriviaQA (n=1000 each), 3 models (7–8B) | | Key result | AUROC 0.820 vs Semantic AU 0.793, at 1/11 the cost |

*Based on arXiv:2605.05166. All figures quoted from the paper. Scope: closed-book short-answer factual QA; long-form generation and multi-step reasoning chains are outside these conclusions.*