Predicting LLM Answer Correctness from Uncertainty Fingerprints: AUROC 0.807 Using Only the First 300 Tokens

> Core claim: Grünefeld et al. (2026) treat an LLM's chain-of-thought like a mystery novel — reading not the content, but the *shape* of its uncertainty signals. Correct and incorrect reasoning chains have distinctly different uncertainty profiles: correct ones look like a steep slide (uncertainty drops fast), incorrect ones like a gentle ramp (hesitant, wavering). Most strikingly, looking at the uncertainty fingerprint of just the first 300 tokens predicts whether the final answer is correct with AUROC 0.801. If this holds, we don't need to wait for a 2000-token answer to know it's wrong — we can terminate early and save ~85% of the compute.

1. An Overlooked Signal: The Reasoning Chain's "Heartbeat"

Think of watching someone solve a math problem:

• The confident solver: glances at the problem, starts writing, and grows steadily more certain
• The lost solver: hesitates, erases, rewrites — uncertainty keeps fluctuating

The authors show LLM reasoning chains exhibit a similar "heartbeat" — token-level uncertainty signals \(U(t)\) (entropy/variance/confidence of the next-token distribution at each position \(t\)).

Rather than feeding the whole curve to a classifier, they extract shape features:

| Feature | Meaning | Correct traces | Incorrect traces | |:---|:---|:---|:---| | Slope | Speed of uncertainty decline | Steep (fast confidence) | Flat (lingering doubt) | | Linearity | Smoothness of the decline | Low (non-linear) | High (mechanical decline) | | Area under curve | Total uncertainty | Lower | Higher | | Peak position | Most uncertain moment | Early (hard parts resolved quickly) | Scattered or late |

> Key finding: Correct traces show a steep, non-linear uncertainty drop; incorrect traces show a slow, linear, mechanical one.

2. Predictive Power: Correctness Visible Within 300 Tokens

Across five LMs on GSM8K and ProntoQA:

• Full-trajectory AUROC: ~0.80 across models, up to 0.807
• First 300 tokens: AUROC 0.801
• First 100 tokens: ~0.75

Since 300 tokens is only 15–30% of a typical trace, AUROC barely degrades — errors are "written on the model's face" early in generation.

3. Correct vs. Incorrect Fingerprints

Correct traces: steep decline, non-linear (natural "thinking bumps" at key steps), early uncertainty peaks — like an experienced doctor who quickly forms a diagnostic direction, verifies a few hypotheses, then concludes confidently.

Incorrect traces: gradual decline, high linearity ("pretending to think" — mechanically filling tokens), persistently high uncertainty — like a student bluffing through a solution and guessing at the end.

4. Why It Matters

| Scenario | Current practice | With uncertainty fingerprints | |:---|:---|:---| | Test-time scaling | Generate full trace, then judge | Judge after 300 tokens | | Self-consistency | Generate 8–16 full traces | Prune clearly wrong ones early | | RLVR training | Reward from full traces | Detect zero-advantage samples early |

If wrong traces can be culled at 300 tokens with ~80% accuracy, test-time compute could be cut by 70–85%.

The post also connects this to earlier discussed work, forming a unified picture: 1. High-entropy tokens are key decision points in reasoning 2. Whole-trace entropy statistics can predict reward (POISE-style probes) 3. But entropy statistics are coarse — the uncertainty profile is the high-resolution fingerprint 4. Using it to prune early avoids wasting compute on doomed traces

This reframes reasoning itself: reasoning is not the accumulation of content but the management of uncertainty. Good reasoning = steep uncertainty decline, not long CoT.

5. The Post's Bet

The author bets that by end of 2026, "uncertainty profiles" will become a standard diagnostic in LLM reasoning systems, with test-time scaling frameworks embedding early error-detection modules that decide after 20–30% of tokens whether to continue, backtrack, or give up. Rationale: AUROC 0.807 is deployable without extra model training; 70–85% compute savings are an engineering goldmine; results generalize across 5 models and 2 tasks; and implementation requires only logging token entropy and computing a few statistics.

6. Limitations and Open Questions

• Task generalization: validated only on GSM8K (math) and ProntoQA (logic). Does it hold for open-ended writing or code generation?
• Scale effects: do larger models "fake confidence" (low uncertainty but wrong)?
• RL integration: could the profile serve as a real-time RLVR reward signal?
• Dynamic generation: continue / backtrack to the last uncertainty peak / abandon and say "I don't know"?

Paper Details

| Item | Content | |:---|:---| | Title | Tracing Uncertainty in Language Model "Reasoning" | | Authors | Nils Grünefeld, Bertram Højer, Philipp Mondorf, Barbara Plank, Anna Rogers, Christian Hardmeier, Stefan Heinrich, Jes Frellsen | | Affiliations | IT University of Copenhagen, DTU, University of Copenhagen, etc. | | arXiv ID | 2605.07776 | | Date | 2026-05-08 | | Contributions | Uncertainty trajectory profiles; shape-feature-based correctness prediction (AUROC 0.807); early detection at 300 tokens (AUROC 0.801); qualitative differences between correct/incorrect traces |

Takeaway: An LLM's reasoning process is not a black box — its "heartbeat," the uncertainty signal, can be read, analyzed, and used to predict success before the answer is even written.