Wrong Prediction, Right Answer: A Two-Parameter Fix Reveals LLM Readout Bottlenecks

*English adaptation of a zhichai.net forum post discussing the paper "Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores".*

An Awkward Scenario

Ask a large language model a logical reasoning question:

> All A are B. All B are C. All C are D. X is A. Is X a D?

The model answers: "Unknown".

You sigh, mark an X on your evaluation sheet: insufficient reasoning ability.

But wait—is the model really unable to reason? Or does it reason fine but fail to say so?

A recent paper by Qiyao Yan, Chenpeng Wang, and Liangming Pan of Peking University, *Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores*, gives an answer that makes you sit up: in many cases the model has already computed the correct answer internally, but structural bias in the output layer suppresses it into "unknown".

Even more striking—tuning just two parameters, using 25 unlabeled samples, can recover the suppressed correct answer.

Core Finding: The Readout Gap

The researchers designed a three-stage diagnostic protocol, tested on the Qwen3.5 model family across four reasoning tasks (controlled logic, ProofWriter, ANLI, FOLIO).

• Stage 1: Probe reading. Train a linear probe (logistic regression) on hidden states before the model answers, to see if the correct answer can be decoded from internal representations.
• Stage 2: Native scoring. Compute sequence scores for candidate answers using the model's own output layer (sum of teacher-forced log-probabilities), and see what the model itself says.
• Stage 3: Offset correction. Add a two-parameter global offset to sequence scores and see whether correct answers can be rescued.

The results are striking. For Qwen3.5-9B:

| Readout method | In-domain accuracy | Lexical OOD accuracy | |---------|-----------|-------------| | Answer-slot probe | 0.972 | 0.830 | | Same-position label logits | 0.573 | 0.574 | | Full sequence score | 0.333 | 0.333 |

0.333 is chance for three-way classification. In other words, the model's sequence scores have completely collapsed—while the probe reads out up to 97% accuracy from the hidden states.

The answer exists inside the model; the output layer just crushes it.

The researchers call this the "readout gap": hidden-state probes successfully decode the correct answer while native sequence scoring collapses entirely. The model doesn't fail to reason—it fails to *express* the answer it computed.

Why It Collapses: The Reign of Structural Bias

To understand the collapse, the paper decomposes the output score additively:

\[s_y(x) = \tilde{s}_y(x) + \beta_y\]
• \(\tilde{s}_y(x)\) is the task-relevant signal—the model's instance-level reasoning result for this question
• \(\beta_y\) is the structural bias—a global preference for certain labels from pretraining frequency priors

The problem: \(\beta_y\) vastly outweighs \(\tilde{s}_y(x)\).

Concretely, Qwen3.5-9B predicts "unknown" on 999 out of 1000 lexical-OOD questions. It's not that the model can't compute the answer—the "unknown" token is extremely frequent in pretraining corpora, and instruction tuning and safety alignment further amplify the preference for uncertainty markers. The output layer has a built-in "safe-word preference" pushing all uncertain cases toward "unknown".

It's like a student who fills in "C" on every uncertain exam question—not because they don't know, but because the habit of "picking C" overwhelms the actual reasoning result.

More critically: this bias exists even before instruction tuning. The researchers examined the Qwen3.5 Base checkpoint (without instruction fine-tuning) and found the same collapse pattern. The bias originates in pretraining frequency priors for structural tokens; instruction tuning only amplifies it.

The Two-Parameter Fix: A 25-Sample Miracle

Now for the best part. If the collapse is caused by a global label-level bias \(\beta_y\), can a simple offset cancel it out?

The fix is minimal: add an offset \(c_y\) to each candidate label \(y\), predicting with

\[\hat{y}_c(x) = \arg\max_y [s_y(x) + c_y]\]

With three labels (true/false/unknown), one offset is fixed at 0 (uniform shifts don't change argmax), leaving two free parameters.

How are they chosen? No labels needed. Given an unlabeled in-domain score set \(D_{id}\), grid search finds the offsets making the predicted distribution closest to a uniform prior (since the evaluation set is label-balanced).

That's all. No gradient descent, no fine-tuning, no labels. Two parameters, 25 unlabeled samples.

Results:

| Task | Model | Before | 25-sample fix | 1000-sample fix | |-----|------|-------|-----------|-------------| | Controlled logic OOD | Qwen3.5-4B | 0.333 | 0.549 | 0.570 | | Controlled logic OOD | Qwen3.5-9B | 0.333 | 0.580 | 0.602 | | ProofWriter | Qwen3.5-4B | 0.333 | 0.641 | 0.644 | | ProofWriter | Qwen3.5-9B | 0.334 | 0.675 | 0.677 | | ANLI R2 | Qwen3.5-4B | 0.445 | 0.559 | 0.574 |

The gap between 25 and 1000 samples never exceeds 0.022. The fix isn't "learning" anything new—it's fitting a threshold offset that already exists. The model's reasoning ability was always there, suppressed by a global bias.

And it transfers across models: it works equally well on OLMo-2-1B and Llama-3.1-8B.

Three Control Experiments: No Shortcuts

Skeptical questions: is the fix exploiting shallow lexical matching? Is it merely aligning label distributions? Three rigorous controls:

1. TF-IDF miss slice: keeping only samples a bag-of-words classifier gets wrong (excluding shallow lexical matches), corrected accuracy on ProofWriter still holds at 0.622 and 0.643.

2. Label-count permutation baseline: keeping the corrected label distribution fixed while randomly permuting per-prediction labels. If the fix only aligned label distributions, permutation should not hurt. The permutation baseline is far below the corrected accuracy (gaps of +0.287 and +0.305), proving the fix aligns instance-level answers.

3. Few-shot subsampling: fitting offsets with 25 to 500 samples. If the fix were "learning" a new task mapping, fewer samples should hurt. In practice 25 samples is nearly saturated, with 30 subsampling runs all positive.

These controls rule out all shortcut explanations. The correction recovers instance-level reasoning structure that already exists in the model.

Boundary Conditions: When the Fix Fails

The paper honestly delineates limits:

1. Near-ceiling rows: Qwen3.5-9B on ANLI already scores 0.638 natively; no collapsible structure to recover, so no gain.

2. Models without collapsed rankings: Pythia-410M and Pythia-12B show no recovery on ProofWriter (p=0.103 and p=0.462). Their hidden rankings contain no recoverable structure—they may genuinely be unable to reason.

3. Wrong prior: if the label prior \(\pi\) is set incorrectly, gains shrink but stay positive. Wrong priors degrade gradually, they don't invert.

These boundaries sharpen the conclusion: the fix works only when probes show the answer is encoded AND native predictions are collapsed. It's not a panacea—it's a precise diagnostic tool.

Why CoT Works: A Mechanistic Explanation

The paper offers a mechanistic account of chain-of-thought prompting—one of its deepest insights.

Single-token answers ("Answer: true/false/unknown") concentrate the entire reasoning burden into one unembedding step—the model must compress multi-step reasoning into a single token's logits. That's exactly where structural bias \(\beta_y\) rules.

CoT distributes reasoning across many intermediate tokens and hidden states. Each intermediate step carries a smaller representational burden and never passes through that "bottleneck unembedding". The reasoning signal accumulates across positions instead of being crushed at one point.

The paper doesn't directly verify this hypothesis (the authors call it speculative), but it predicts: collapsing should shrink when moving from single-token answers to free generation. This matches known empirical findings—CoT does improve reasoning tasks—but no one had previously given such a clean mechanistic explanation.

Implications for Evaluation Practice

The paper delivers a direct blow to LLM evaluation practice:

> Before concluding "the model lacks reasoning ability", check the prediction histogram. If predictions have collapsed onto a single label, a lightweight unlabeled offset correction can reveal whether capability is masked or genuinely missing.

Current leaderboards may be punishing some models—not for poor reasoning, but for poor default calibration. Model A scores 0.33 on a benchmark and model B scores 0.55; after two-parameter correction A jumps to 0.60 while B stays put—A's reasoning is actually stronger; it just has a bug in its expression layer.

This belongs to the same family as "judgment-gate decoupling": a model's internal judgment and external action are two separate modules. Internal judgment is correct (hidden states encode the answer), but the action gate (unembedding matrix + sequence aggregation) doesn't consult the judgment module and outputs according to structural bias.

The fix only targets the gate—no retraining, no change to the inference path, just two parameters to cancel the gate's bias.

Practical Engineering Value

For engineering practice, the paper offers actionable insights:

1. Diagnose before fixing. Before spending resources on RLHF or fine-tuning, run a probe + offset diagnostic. High probe accuracy with collapsed sequence scores means the problem is in the output layer; collapsed probes mean the problem is in representations—two problems with entirely different fix strategies.

2. The power of unlabeled calibration. The two-parameter offset needs only 25 unlabeled samples. For any forced-choice task (classification, entailment, QA), this is a near-zero-cost baseline calibration method.

3. Prediction histograms are a free signal. If a model predicts the same label for 999/1000 samples on a balanced dataset, that itself is a bug signal—no probe training needed to detect it.

4. A mechanistic basis for CoT. If single-token answers collapse but CoT doesn't, the problem is the unembedding bottleneck, not reasoning ability. This gives CoT a deeper framing than "prompt engineering"—it's an architecture-level strategy to bypass structural output-layer bias.

Personal Reflection: The Universality of Expression Bottlenecks

This paper raises a broader question: is the expression bottleneck unique to LLMs, or a common ailment of all information systems?

Humans have something similar—you know the answer but write it wrong on the exam. Psychology calls this "expression failure", distinct from "competence deficit". But human expression failures are sporadic, while LLM expression failures are systematic—the same model repeatedly collapses to the same label on the same class of tasks.

The root of this systematic collapse is that the unembedding matrix is static—the same linear transformation processes every input. Whatever the hidden state \(h^a(x)\) encodes, it must pass through the same \(W^T h + b\) projection. If certain columns of \(W\) (corresponding to tokens like "unknown") have especially large norms or high biases, they dominate the argmax.

This is another facet of the same problem as the "softmax bottleneck" (Yang et al., 2017): the output layer's expressiveness is limited by the linear projection onto vocabulary dimensions. Hidden states can be arbitrarily high-dimensional manifolds, but they all pass through this fixed bottleneck.

The paper's fix—a global offset—can only cancel label-level constant bias. The authors honestly note it cannot repair instance-level distortions (like surface-form competition). It's like giving a nearsighted person reading glasses—distance vision clears, but near-field astigmatism remains. Still, it tells us: the eyesight is there; the lens just needs adjusting.

Paper Information

definitely worth bookmarking:

• Title: Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
• Authors: Qiyao Yan, Chenpeng Wang, Liangming Pan (Peking University / Yixin AI Lab / Beijing Academy of Artificial Intelligence)
• arXiv: 2608.31068
• Published: August 31, 2026
• Paper link: https://arxiv.org/abs/2608.31068
• HTML full text: https://arxiv.org/html/2608.31068v1

*A reminder for everyone doing LLM evaluation: before saying "the model can't", confirm whether the model "can but can't say it". Sometimes the deepest fix isn't retraining—it's tuning two parameters.*

*Conceptual lineage: this post belongs to the "judgment-gate decoupling" series—the model's internal judgment is correct, but the output layer (gate) is dominated by structural bias and never consults the judgment module. Fixes need only target the gate, not the model itself. Related posts in this series: eval-awareness dual personas, PoP layer-wise hesitation, omission blindness.*