Uncertainty Fingerprints Predict LLM Answer Correctness: AUROC 0.807 from First 300 Tokens
Core claim: Grünefeld et al. (2026) treat an LLM's chain-of-thought like a mystery novel—not reading the content, but the *shape* of its uncertainty signals. They found that correct and incorrect reasoning chains have completely different uncertainty profiles: correct ones look like steep slides (uncertainty drops quickly), while incorrect ones look like gentle ramps (hesitant, oscillating). Even more striking: just the first 300 tokens of uncertainty fingerprint predict the final answer's correctness with AUROC 0.801. If this holds, we don't need to wait for a 2000-token answer to know it's wrong—we can terminate early in the first paragraph and save ~85% of compute.
Key points
- Token-level uncertainty as a signal: At each position \(t\), the entropy/variance/confidence over the next-token distribution forms an uncertainty curve \(U(t)\).
- Profile features extracted: slope (speed of confidence gain), linearity (smoothness of decline), area under the curve, and peak position.
- Correct traces: steep, nonlinear decline with early peaks—like an experienced doctor forming a diagnosis quickly, with natural "thinking bumps" along the way.
- Incorrect traces: flat, linear, mechanical decline with persistently high uncertainty—like a student bluffing, writing many words while unsure of every step.
- Task generalization: validated only on GSM8K (math) and ProntoQA (logic); does it hold for open-ended tasks like code generation or creative writing?
- Model scale: do larger models "fake confidence" (low uncertainty despite errors)?
- RL integration: could the profile serve as a real-time reward signal in RLVR—positive reward for steep declines, negative for flat ones?
- Dynamic strategies: backtrack to the nearest uncertainty peak when the profile looks wrong; give up after repeated failures.
Predictive power
Experiments across 5 LMs on GSM8K and ProntoQA:
| Tokens used | AUROC | |:---:|:---:| | Full trajectory (~1000-2000 tokens) | 0.807 | | First 300 tokens | 0.801 | | First 100 tokens | ~0.75 |
300 tokens is only 15-30% of the trajectory, yet AUROC barely drops—the error is "written on the face" of the reasoning early on.
Why it matters
| Scenario | Current practice | With uncertainty fingerprints | |:---|:---|:---| | Test-time scaling | Generate full traces, then judge | Judge after 300 tokens | | Self-consistency | Generate 8-16 full traces | Prune clearly wrong ones early | | RLVR training | Reward from full traces | Detect zero-advantage samples early |
Early detection at 80% accuracy could save 70-85% of test-time compute. It also connects to prior findings: high-entropy tokens as key decision points, entropy statistics predicting reward—the uncertainty profile unifies these into an actionable, high-resolution signal.
A redefinition of "reasoning"
| Traditional view | New view | |:---|:---| | Reasoning = writing more steps | Reasoning = reducing uncertainty | | Long CoT = good reasoning | Steep uncertainty decline = good reasoning | | Judge correctness by final answer | Judge correctness by the uncertainty fingerprint |
Limitations and open questions
Paper details
| Item | Content | |:---|:---| | Title | Tracing Uncertainty in Language Model "Reasoning" | | Authors | Nils Grünefeld, Bertram Højer, Philipp Mondorf, Barbara Plank, Anna Rogers, Christian Hardmeier, Stefan Heinrich, Jes Frellsen | | Institutions | IT University of Copenhagen, DTU, University of Copenhagen, etc. | | arXiv ID | 2605.07776 | | Date | 2026-05-08 | | Core contributions | Uncertainty trajectory profiles; shape-feature-based correctness prediction (AUROC 0.807); early detection (300 tokens, AUROC 0.801); qualitative differences between correct/incorrect traces |