When Are Likely Answers Right? Sequence Probability vs. Correctness in LLMs
> Paper: When are likely answers right? On Sequence Probability and Correctness in LLMs > Authors: Johannes Zenn, Jonas Geiping > arXiv: 2606.27359 > Published: 2026-06-25
This is a full English translation of a Chinese forum post that explains the paper in an accessible, Feynman-style narrative.
🍷 The Banquet Illusion
Imagine a dinner party. A woman in a red dress stands in the corner, drawing everyone's gaze — not because she is the most beautiful, but because she is the most *conspicuous*. In the world of LLMs, "conspicuous" means high probability. The higher sequence probability a model assigns to an answer, the more "natural" and "fluent" it seems given the model's training data.
The fundamental question this paper asks: Is the most conspicuous answer necessarily the correct one? Zenn and Geiping systematically study the real relationship between a model's "conspicuousness" (probability) and correctness — and the findings may make you rethink what we are actually tuning when we adjust temperature, beam search, or top-p sampling.
🧭 Decoding Methods: From Local to Global Probability Hunters
Decoding is how an LLM picks its output: at each step, the model assigns a probability to every possible next token, and a decoding strategy decides how to choose.
Local decoding: one step at a time
- Low-Temperature Sampling (LTS): the most conservative option — temperature controls how much the model favors high-probability tokens.
- Top-k / Top-p sampling: only consider the top k tokens, or the smallest set whose cumulative probability reaches p.
- ε-sampling: any token above a small threshold ε can be selected.
- Beam Search tracks K candidate paths at once (K=1 reduces to greedy decoding). But beam search famously produces repetitive, bland, lower-quality text — like walking the "safest" path through a mall and always passing the same stores.
- Best-of-N (BoN) generates N complete answers, then picks the highest-probability one. As N → ∞ it approximates the mode.
- Power Sampling sharpens the whole sequence distribution without changing local choices; the parameter α controls sharpness. As α → ∞, the optimizer collapses onto the distribution's mode.
- 🟩 Green (+/+): higher probability, higher accuracy
- 🟨 Yellow (+/-): higher probability, *lower* accuracy — a dangerous trap
- 🟦 Blue (-/+): lower probability, higher accuracy — an undervalued treasure
- 🟥 Red (-/-): both lower
- MATH500: near-perfect positive correlation (Pearson r = +0.96) — the model genuinely "senses" whether its answer is right.
- GPQA, HumanEval, MedQA, MMLU: moderate positive correlation.
- IFEval: negative correlation — the more "confident" the answer, the more likely it is wrong, possibly because base models find instruction formats unfamiliar and assign high probability to fluent-but-wrongly-formatted answers.
- Don't blindly trust global methods: beam search and power sampling find high-probability sequences that aren't necessarily better; on many tasks simple low-temperature sampling suffices.
- Hyperparameter tuning is necessary — no universal optimum exists.
- Power sampling is safe on MATH: math is the one task where probability and correctness align tightly.
- Plain self-consistency works better with uniform voting than probability-weighted voting, since per-question probability barely correlates with correctness.
- Power self-consistency (PSC) helps on MATH but is unstable elsewhere.
- Zenn J., Geiping J. (2026). When are likely answers right? On Sequence Probability and Correctness in LLMs. *arXiv preprint arXiv:2606.27359*.
- Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. *ACL*.
- Holtzman, A., et al. (2020). The Curious Case of Neural Text Degeneration. *ICLR*.
- Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *ICLR*.
- Ji, X., et al. (2026). Scalable Power Sampling for Large Language Models. *arXiv*.
- Karan, M., & Du, L. (2025). Power Sampling for Language Models. *arXiv*.
- Azizi, S., et al. (2026). Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning. *arXiv*.
- Yang, A., et al. (2025). Qwen3 Technical Report. *arXiv*.
All of these make local decisions — each step picks what looks best, but no one guarantees the combination is globally optimal.
Global decoding: optimizing the whole sequence
The variational view
The paper gives an elegant mathematical framing: all global decoding methods optimize a variational objective balancing expected sequence probability against output diversity. The power distribution p_α is the analytic solution to this optimization problem, which explains why large-N BoN behaves like power sampling at large α.
🔬 Four Levels of Experiments
The authors ran a systematic correlation analysis: 8 decoding methods, 14 models (Qwen2.5, Qwen3, OLMo3 families), 6 benchmarks (MATH500, GPQA, HumanEval, IFEval, MedQA, MMLU), at four levels of granularity:
1️⃣ Across-Method
Taking LTS (α=4) as a baseline, each method is classified:
Finding: MATH500 shows the most green cells — probability and correctness align well on math. But on MMLU, HumanEval, MedQA, GPQA, and IFEval, yellow and blue cells abound: no method reliably beats the LTS baseline, and higher-probability answers are not necessarily more correct.
2️⃣ Within-Method
Turning up hyperparameters does produce higher-probability sequences — but accuracy goes up about half the time and down the other half. Like turning up a stereo's volume: louder, but not necessarily better sound. In practice, you cannot assume "more conservative = more correct"; hyperparameters must be tuned per method, model, and dataset.
3️⃣ Within-Dataset (the most striking finding)
Binning samples by probability into 10 bins and measuring per-bin accuracy:
Crucially: post-trained models are almost always positively correlated, while base models are erratic and often negative. Post-training (SFT/RLHF) improves not just capability but also calibration — the model's judgment of its own correctness.
4️⃣ Within-Sample
Sampling the same prompt 32 times: correlation coefficients are distributed symmetrically around zero. For a single question, probability tells you nothing about which answer is right. One exception: when the model's baseline accuracy on that question is already high (e.g., 8 of 10 correct), the probability–correctness correlation strengthens — the probability signal is only reliable when the model is already "strong enough."
🎯 Core Conclusions: The Usability Boundary of Probability
1. Across samples (within a dataset): probability *is* a useful signal — but its strength depends on task type and model quality. 2. Across methods / hyperparameters: probability is *not* a useful selection criterion. 3. Across repeated samples of one prompt: probability is *nearly useless*. 4. Key precondition: the reliability of the probability signal is positively tied to the model's baseline accuracy on the task. Only a model that is already strong can use probability as a self-improvement cue; a weak model will reinforce its own errors.
💡 Practical Implications
Decoding strategy
Self-consistency
Verifier-free self-improvement
Approaches that use probability itself as a reward signal (e.g., self-distilling high-probability samples) get a cold shower here: the signal is only reliable once the model's accuracy is already high. Otherwise the model keeps reinforcing wrong patterns — like letting a failing student grade their own exam and confidently check the wrong answers.
🌌 Philosophical Coda
Probability is the model's "intuition" — statistical regularities learned from training data — and intuition can deceive, especially on unseen tasks. The paper's contribution is a more precise map: trust probability as a filter in mathematical reasoning; beware overconfident answers in instruction-following; and never assume "higher probability = better" when choosing decoding methods. Probability is a tool — and the point of a tool is knowing when to use it and when to put it down.