When Are Likely Answers Right? Sequence Probability vs. Correctness in LLMs

> Paper: When are likely answers right? On Sequence Probability and Correctness in LLMs > Authors: Johannes Zenn, Jonas Geiping > arXiv: 2606.27359

This post is an in-depth Chinese-to-English breakdown of a paper asking a fundamental question: is a high-probability answer from an LLM also a correct one? Like the most eye-catching guest at a dinner party, high probability means salience — not necessarily truth.

Background: Decoding Methods

The paper distinguishes local decoding strategies (low-temperature sampling, top-k, top-p, ε-sampling) that pick tokens step-by-step, from global decoding strategies that aim for high-probability whole sequences:

• Beam search: tracks K paths simultaneously; notoriously produces repetitive, bland text despite finding high-probability sequences.
• Best-of-N (BoN): generates N full answers, selects the highest-probability one.
• Power sampling: sharpens the entire sequence distribution with exponent α; as α→∞ it converges to the distribution's mode. The paper frames all global decoding methods as optimizing a variational objective balancing expected sequence probability and output diversity.

Four Levels of Correlation Analysis

The authors ran experiments across 8 decoding methods, 14 models (Qwen2.5, Qwen3, OLMo3), and 6 benchmarks (MATH500, GPQA, HumanEval, IFEval, MedQA, MMLU).

1. Across-Method

Compared against low-temperature sampling as a baseline, no method consistently achieved both higher probability and higher accuracy. MATH500 showed the best agreement; MMLU, HumanEval, MedQA, GPQA, and IFEval showed many cases where higher probability did not mean higher correctness.

2. Within-Method

Increasing hyperparameters reliably raised sequence probability, but accuracy improved only about half the time — you cannot assume "more conservative decoding = more correct."

3. Within-Dataset

The strongest finding. Binning samples by probability:
• MATH500: near-perfect positive correlation (Pearson r = +0.96).
• GPQA, HumanEval, MedQA, MMLU: moderate positive correlation.
• IFEval: negative correlation — more confident answers were more likely wrong, likely due to base models favoring fluent but format-incorrect outputs.

Crucially, post-trained models (SFT/RLHF) show consistent positive correlation, while base models are erratic and often negative — post-training improves calibration, not just capability.

4. Within-Sample

For 32 repeated samples of the same prompt, correlation coefficients cluster around zero: probability is essentially useless for picking the best answer to a single question. The exception: when the model's baseline accuracy on that question is already high, the probability signal becomes meaningful.

Core Conclusions

1. Across samples (within a dataset): probability is a useful signal, but strength depends on task and model quality. 2. Across methods/hyperparameters: probability is not a reliable selection criterion. 3. Within repeated samples of one prompt: probability is nearly useless. 4. Key precondition: the signal's reliability correlates with the model's baseline accuracy on the task.

Practical Implications

• Decoding: don't blindly trust beam search or power sampling; plain low-temperature sampling is often sufficient. Power sampling is a safe bet mainly on math tasks.
• Self-consistency: uniform voting typically beats probability-weighted voting; power self-consistency works on MATH but is unstable elsewhere.
• Verifier-free self-improvement: using probability as a reward signal only works if the model is already reasonably accurate — otherwise the model reinforces its own confident mistakes, like a failing student grading their own exam.

Takeaway

Probability is a multi-faceted signal: informative across samples, near-random within samples, and unreliable for method selection. The paper provides a precise map of when to trust model confidence — and when to put it down.

References

• Zenn J., Geiping J. (2026). When are likely answers right? On Sequence Probability and Correctness in LLMs. *arXiv preprint arXiv:2606.27359*.
• Holtzman, A., et al. (2020). The Curious Case of Neural Text Degeneration. *ICLR*.
• Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *ICLR*.
• Yang, A., et al. (2025). Qwen3 Technical Report. *arXiv*.