Wrong Prediction, Right Answer: A Two-Parameter Fix Reveals LLM Readout Bottleneck

Forum topic · 小凯 · 2026-09-01

Summary

A new paper from Peking University researchers (Yan, Wang, Pan), "Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores" (arXiv:2608.31068), shows that large language models often compute correct reasoning answers internally but fail to express them at the output layer. Across Qwen3.5 models and four reasoning tasks (controlled logic, ProofWriter, ANLI, FOLIO), linear probes on hidden states decoded answers at up to 0.972 accuracy while native sequence scores collapsed to 0.333—random chance—because a large structural prior favoring tokens like "unknown" dominates the unembedding step. The authors decompose sequence scores into task signal plus label-level structural bias and propose a two-parameter additive offset calibrated on just 25 unlabeled samples, restoring substantial accuracy (e.g., ProofWriter 0.333 to 0.675) with no gradients or fine-tuning. Three control experiments rule out shallow lexical matching or mere label-distribution alignment. The paper also offers a mechanistic account of why chain-of-thought prompting works—distributing reasoning away from the single-token unembedding bottleneck—and gives practical guidance for LLM evaluation: check prediction histograms before concluding a model lacks reasoning ability.

Wrong Prediction, Right Answer: The Readout Bottleneck in LLMs

A forum discussion of the paper "Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores" by Qiyao Yan, Chenpeng Wang, and Liangming Pan (Peking University), arXiv:2608.31068.

The Problem

Ask an LLM a simple syllogism (All A are B; All B are C; All C are D; X is A — is X a D?) and it answers "unknown." Is the model failing to reason — or reasoning successfully but unable to *say* the answer? The paper's answer: often the model has already computed the correct answer internally, but a structural bias at the output layer suppresses it.

The Readout Gap

The authors use a three-stage protocol (linear probes on hidden states → native sequence scoring → bias offset) on Qwen3.5 models across four reasoning tasks (controlled logic, ProofWriter, ANLI, FOLIO). For Qwen3.5-9B:

| Readout | In-domain acc. | Lexical OOD acc. | |---|---|---| | Answer-slot probe | 0.972 | 0.830 | | Same-position label logits | 0.573 | 0.574 | | Full sequence scores | 0.333 | 0.333 |

0.333 is chance for three-way classification: sequence scores collapse completely while probes recover up to 97% accuracy. The researchers call this the readout gap.

Why It Collapses

Sequence scores decompose as:

s_y(x) = \tilde{s}_y(x) + \beta_y

where $\tilde{s}_y(x)$ is instance-level task signal and $\beta_y$ is a global structural bias from pretraining token-frequency priors (amplified by instruction tuning). $\beta_y$ dominates: Qwen3.5-9B predicts "unknown" on 999/1000 lexical-OOD items. The same collapse appears in the Base checkpoint, so the bias originates in pretraining.

The Two-Parameter Fix

Since the bias is label-level and additive, a per-label offset suffices:

\hat{y}_c(x) = \arg\max_y [s_y(x) + c_y]

One offset is fixed to 0, leaving two free parameters, calibrated on just 25 unlabeled samples via grid search toward a uniform prediction distribution. No gradients, no fine-tuning, no labels.

| Task | Model | Before | 25 samples | 1000 samples | |---|---|---|---|---| | Controlled logic OOD | Qwen3.5-4B | 0.333 | 0.549 | 0.570 | | Controlled logic OOD | Qwen3.5-9B | 0.333 | 0.580 | 0.602 | | ProofWriter | Qwen3.5-4B | 0.333 | 0.641 | 0.644 | | ProofWriter | Qwen3.5-9B | 0.334 | 0.675 | 0.677 | | ANLI R2 | Qwen3.5-4B | 0.445 | 0.559 | 0.574 |

25 samples and 1000 samples differ by at most 0.022 — the offset fits an existing threshold, not a new task mapping. The fix also transfers to OLMo-2-1B and Llama-3.1-8B.

Control Experiments

1. TF-IDF hard slice: accuracy stays high (0.622, 0.643 on ProofWriter) after removing shallow lexical-match cases. 2. Label-count permutation baseline: shuffling predictions while keeping the label distribution drops accuracy far below the fix (gap +0.287 / +0.305), proving instance-level recovery. 3. Few-shot subsampling: 25 samples already saturate; all 30 subsamples improve.

Boundaries

The fix fails or adds nothing when: the model is near ceiling (Qwen3.5-9B on ANLI, 0.638); models lack recoverable hidden ordering (Pythia-410M/12B, p=0.103 / 0.462); or the uniform prior is wrong (gains shrink but stay positive).

Why CoT Helps: A Mechanistic Account

Single-token answers concentrate all reasoning into one unembedding step — exactly where structural bias $\beta_y$ rules. Chain-of-thought distributes reasoning across many intermediate tokens, bypassing the bottleneck. The authors flag this as speculative but consistent with known CoT gains.

Implications for Evaluation and Engineering

Before claiming a model "can't reason," check the prediction histogram. A collapsed prediction distribution signals suppression, not absence of ability.
Diagnose before fixing: high probe accuracy + collapsed scores = output-layer problem; both collapsed = representation problem. The remedies differ entirely.
Label-free calibration is nearly free: a two-parameter offset on 25 unlabeled samples is a baseline for any forced-choice task.
A model scoring 0.33 raw but 0.60 after the offset may genuinely reason better than one scoring 0.55 raw — leaderboards may be penalizing calibration, not reasoning.

Discussion

The post frames this as an instance of "judgment–gate decoupling": internal judgment is correct, but the output gate (unembedding + sequence aggregation) obeys structural bias without consulting the judgment. Related to the softmax bottleneck (Yang et al., 2017): the static linear projection limits output expressiveness regardless of the richness of hidden states. The global offset only removes label-level constants — instance-level distortions like surface-form competition remain, but the result proves the capability exists and only the "lens" needs adjusting.

Paper Information

Title: Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Authors: Qiyao Yan, Chenpeng Wang, Liangming Pan (Peking University)
arXiv: 2608.31068
Link: https://arxiv.org/abs/2608.31068
HTML: https://arxiv.org/html/2608.31068v1