An Embarrassing Scenario
Ask a large language model a simple logical reasoning question:
> All A are B. All B are C. All C are D. X is A. Is X a D?
The model answers: "Unknown".
You sigh and mark a cross on your evaluation sheet: insufficient reasoning ability.
But wait — can the model really not reason? Or does it reason correctly but cannot express it?
A new paper by Qiyao Yan, Chenpeng Wang, and Liangming Pan of Peking University, *Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores*, gives an answer that makes you sit up: in many cases the model has already computed the correct answer internally, but a structural bias in the output layer crushes it into "unknown".
Even more striking — by tuning just two parameters with 25 unlabeled samples, the suppressed correct answers can be recovered.
Core Finding: The Readout Gap
The researchers designed a three-stage diagnostic protocol, tested on the Qwen3.5 family across four reasoning task types (controlled logic, ProofWriter, ANLI, FOLIO):
1. Probe readout: train a linear probe (logistic regression) on hidden states before the model answers, to see whether the correct answer can be decoded from internal representations. 2. Native scoring: compute sequence scores (sum of teacher-forced log-probabilities) for candidate answers using the model's own output layer. 3. Offset correction: add a two-parameter global offset to the sequence scores and see whether correct answers can be rescued.
The result is shocking. For Qwen3.5-9B:
| Readout method | In-domain accuracy | Lexical OOD accuracy | |---------|-----------|-------------| | Answer-slot probe | 0.972 | 0.830 | | Label logits at same position | 0.573 | 0.574 | | Full sequence score | 0.333 | 0.333 |
0.333 is random guessing for a three-way classification. The model's sequence scores have completely collapsed — while the probe reads out the correct answer from hidden states at up to 97% accuracy.
The answer exists inside the model; the output layer just crushes it.
The researchers call this the "readout gap": hidden-state probes successfully decode the correct answer while native sequence scoring collapses entirely. The model doesn't fail to reason — it fails to *say* what it computed.
Why It Collapses: The Reign of Structural Bias
The paper decomposes the output score additively:
- \(\tilde{s}_y(x)\) is the task-relevant signal — the model's instance-level reasoning result for this question.
- \(\beta_y\) is the structural bias — a global, pretraining-frequency prior over labels.
- Title: Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
- Authors: Qiyao Yan, Chenpeng Wang, Liangming Pan (Peking University / Yixin AI Lab / Beijing Academy of Artificial Intelligence)
- arXiv: 2608.31068
- Published: August 31, 2026
- Paper link: https://arxiv.org/abs/2608.31068
- Full HTML: https://arxiv.org/html/2608.31068v1
The problem: \(\beta_y\) vastly exceeds \(\tilde{s}_y(x)\).
Specifically, Qwen3.5-9B predicts "unknown" for 999 out of 1000 lexical-OOD questions. Not because the model can't compute the answer, but because the "unknown" token is extremely frequent in the pretraining corpus, and instruction tuning plus safety alignment further amplify the preference for uncertainty markers. The output layer has a built-in "safe-word preference" pushing everything uncertain toward "unknown".
It's like a student who fills in "C" for every uncertain exam question — not because they don't know, but because the "pick C" habit is so strong it overwhelms the actual reasoning result.
Crucially, this bias exists even before instruction tuning. The authors checked the Qwen3.5 Base checkpoint and found the same collapse pattern. The bias stems from pretraining frequency priors over structural tokens; instruction tuning only amplifies it.
The Two-Parameter Fix: A Miracle from 25 Samples
If the collapse is caused by a global label-level bias \(\beta_y\), can a simple offset cancel it?
The correction scheme is minimal: add an offset \(c_y\) per candidate label, predicting
With three labels (true/false/unknown) and one offset fixed at 0 (uniform shifts don't change argmax), only two free parameters remain.
How are they chosen? No labels needed. Just an unlabeled in-domain score set \(D_{id}\), with grid search finding the offsets that make the predicted distribution closest to a uniform prior (since the evaluation set is label-balanced).
That's all. No gradient descent, no fine-tuning, no labels. Two parameters, 25 unlabeled samples.
Results:
| Task | Model | Before | 25-sample fix | 1000-sample fix | |-----|------|-------|-----------|-------------| | Controlled logic OOD | Qwen3.5-4B | 0.333 | 0.549 | 0.570 | | Controlled logic OOD | Qwen3.5-9B | 0.333 | 0.580 | 0.602 | | ProofWriter | Qwen3.5-4B | 0.333 | 0.641 | 0.644 | | ProofWriter | Qwen3.5-9B | 0.334 | 0.675 | 0.677 | | ANLI R2 | Qwen3.5-4B | 0.445 | 0.559 | 0.574 |
The gap between 25 and 1000 samples never exceeds 0.022. This shows the fix isn't "learning" anything new — it's merely fitting a threshold offset that already exists. The model's reasoning ability was there all along, suppressed by a global bias.
The method also transfers across models: it works on OLMo-2-1B and Llama-3.1-8B as well.
Three Control Experiments: No Shortcuts
Three strict controls rule out trivial explanations:
1. TF-IDF failure slice: keeping only samples a bag-of-words classifier gets wrong (excluding shallow lexical matching). Corrected accuracy still holds at 0.622 and 0.643 on ProofWriter.
2. Label-count permutation baseline: keeping the corrected predictions' label distribution fixed but randomly shuffling per-sample labels. If the fix only aligned label distributions, shuffling shouldn't matter. The permutation baseline falls far below the corrected accuracy (gaps of +0.287 and +0.305), proving the fix aligns instance-level answers.
3. Few-shot subsampling: fitting offsets on 25–500 samples. If the fix were learning a new task mapping, fewer samples would hurt. In practice, 25 samples nearly saturate, with all 30 subsampling runs positive.
Boundary Conditions: When the Fix Fails
The paper honestly delineates limits:
1. Near-ceiling rows: Qwen3.5-9B on ANLI starts at 0.638 — no recoverable structure, no offset gain.
2. Models without collapsed ordering: Pythia-410M and Pythia-12B show no recovery on ProofWriter (p=0.103 and p=0.462). Their hidden orderings don't contain recoverable structure — they may genuinely not reason.
3. Wrong prior: if the label prior \(\pi\) is mis-set, gains shrink but stay positive; degrading gradually, not reversing.
The correction works only when probes show the answer is encoded and the raw prediction has collapsed. It's not a panacea — it's a precise diagnostic tool.
Why CoT Works: A Mechanistic Explanation
The paper offers a mechanism-level account of CoT prompting — perhaps its deepest insight.
Single-token answers ("Answer: true/false/unknown") concentrate the entire reasoning burden on one unembedding step — the model must compress multi-step reasoning into a single token's logits. That step is exactly where structural bias \(\beta_y\) rules.
CoT spreads reasoning across multiple intermediate tokens and hidden states. Each step carries a smaller representational burden and never passes through the "bottleneck unembedding". The reasoning signal accumulates across positions instead of being crushed at one point.
The authors flag this as speculative, but it predicts: the collapse should shrink when moving from single-token answers to free generation — consistent with known CoT gains, but now with a much cleaner mechanistic story.
Implications for Evaluation Practice
> Before concluding a model "lacks reasoning ability", check the prediction histogram. If predictions collapse to a single label, a lightweight unlabeled offset correction can reveal whether the capability is masked or genuinely missing.
Current leaderboards may be punishing models not for poor reasoning but for poor default calibration. Model A scores 0.33 on a benchmark, model B scores 0.55 — but after the two-parameter fix, A rises to 0.60 while B stays put. A's reasoning is actually stronger; its expression layer is just buggy.
This belongs to the same family as "judgment-gate decoupling": internal judgment and external action are separate modules. Internal judgment is correct (hidden states encode the answer), but the action gate (unembedding matrix + sequence aggregation) doesn't consult the judgment module and outputs according to structural bias.
The fix targets the gate only — no retraining, no change to the reasoning path, just two parameters canceling the gate's bias.
Practical Engineering Value
1. Diagnose before fixing. Before spending resources on RLHF or fine-tuning, run a probe + offset diagnosis. High probe accuracy with collapsed sequence scores means the problem is in the output layer; collapsed probes mean the problem is in representations — entirely different remediation strategies.
2. The power of unlabeled calibration. The two-parameter offset needs just 25 unlabeled samples — a near-zero-cost baseline calibration for any forced-choice task (classification, entailment, QA).
3. Prediction histograms are a free signal. If a model predicts the same label for 999/1000 samples on balanced data, that's a bug signal by itself — no probe training required.
4. A mechanistic basis for CoT. If single-token answers collapse but CoT doesn't, the problem is the unembedding bottleneck, not reasoning ability. CoT is better understood as an architecture-level strategy for bypassing structural output bias, not mere prompt engineering.
Personal Reflection: The Universality of Expression Bottlenecks
Is the expression bottleneck unique to LLMs, or a disease of all information systems? Humans have it too — you know the answer but write it wrong on an exam. Psychology distinguishes "expression failure" from "competence deficit". But human expression failures are sporadic; LLM expression failures are systematic — the same model repeatedly collapsing to the same label on the same task class.
The root cause is that the unembedding matrix is static — the same linear transform processes all inputs. Whatever the hidden state \(h(x)\) encodes, it must pass through the same \(W^T h + b\) projection. If some columns of \(W\) (tokens like "unknown") have unusually large norms or high biases, they dominate the argmax.
This is a different facet of the same problem as the "softmax bottleneck" (Yang et al., 2017): the output layer's expressivity is limited by the linear projection onto vocabulary dimensions. Hidden states can live on arbitrary high-dimensional manifolds, but they all squeeze through this fixed bottleneck.
The global-offset fix only cancels label-level constant bias; the authors honestly note it cannot repair instance-level distortions (like surface-form competition). It's like giving a nearsighted person reading glasses — distance vision is fixed, but astigmatism up close remains. At least it tells us: the eyesight is there; the lens just needs adjusting.
Paper Information
*A reminder for everyone doing LLM evaluation: before saying a model "can't", confirm whether it "can but can't say". Sometimes the deepest fix isn't retraining — it's adjusting two parameters.*
*Concept lineage: this post belongs to the "judgment-gate decoupling" series — internal judgment is correct, but the output gate is ruled by structural bias and never consults the judgment module. Fixing the gate suffices; the model body is untouched. Related: eval-awareness dual personas, PoP inter-layer hesitation, omission blindness.*