Red Pen and Check Marks: Why a Teacher Holding the Answer Key Still Can't Find the Student's Mistake
*This is the first article in a three-part series on "teaching signals" for AI reasoning, originally posted in Chinese on zhichai.net. It analyzes "A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning" (arXiv:2610.13744, NeurIPS 2026 FLLMPT Workshop oral), by Maniparambil, Rai, Verma, O'Connor, and Reid (Fin.ai and Dublin City University).*
Two Kinds of Teachers
Imagine two teachers grading your exam. The first checks only your final answer: check mark or cross, red pen never leaving the score column. You know you got three problems wrong—but not which step went wrong. That's not teaching; it's a verdict. The second teacher circles your sentences and writes in the margin: "the subject shifted here," "this metaphor contradicts the mood above—try again." Her red pen lands on specific spots with specific reasons. You may disagree, but you know what to fix.
This paper is about exactly this—except the protagonist is an AI. A "privileged teacher" that holds the reference answer scores a student model's reasoning chains. Intuition says: it has seen the answer, so it must know where the student went wrong. The experiments show the opposite: it doesn't. It never did. But when the same model performs a different action—not scoring, but critiquing—the same privileged information suddenly grows eyes.
Background: How Reasoning Models Are Trained
Modern reasoning models are typically trained with RLVR (reinforcement learning with verifiable rewards): the model generates a full reasoning chain, a verifier checks only the final answer, and the reward is 1 or 0. A single scalar over a chain of hundreds of tokens is extremely sparse—if only step three was wrong, the reward says only "total failure," not "the lesion is at step three."
GRPO (Group Relative Policy Optimization, from DeepSeekMath) softens the sparsity: sample G trajectories per problem, compare them within the group, reward above-average ones and penalize below-average ones. But GRPO has a fatal structural blind spot called the all-wrong group: if all G sampled trajectories fail, every advantage is zero and the gradient vanishes. The cruel irony: all-wrong problems sit exactly at the model's capability boundary—they are the most valuable to learn from, and GRPO's teaching signal there is zero.
A natural patch is On-Policy Self-Distillation (OPSD): the same model plays both roles. The student generates a rollout; the teacher, conditioned on the reference answer (privileged information the student can't see), scores each token via KL divergence or per-token probability differences. Hundreds of tokens yield hundreds of dense signals, distilled back into the student. Dense, on-policy—sounds perfect.
The paper's core critique is razor-sharp: density is not credit assignment. Counting ten thousand grains of sand doesn't mean you found the gold nugget.
Finding 1: The Privileged Signal Is Empty Where It Matters
The paper tests signals against gold error spans (error sentences annotated by a judge model with double-review confirmation) using two criteria:
- Localization: the signal at error positions should be significantly larger than at clean positions.
- Correction: the signal should push down the probability of the student's erroneous tokens.
- Oracle critic (GPT-5.2): localizes the gold error in 57%–67% of failed rollouts (K&K 57%, L5-7 62%, L7-10 67%), with >95% of quotes verifiably present in the student text and 100% format compliance.
- Self-critique: 18%–21% localization—weak but directionally right for a 1.7B model, and it scales: Olmo-3-32B reaches 0.47.
- Error-span Z-scores: with oracle critiques, all 18 settings pass both criteria. Qwen3-1.7B: localization Z-scores +0.75, +0.99, +0.78; correction -0.62, -0.90, -0.54—several times stronger than the reference condition.
- Ablations: quote alone keeps localization but weakens correction; reason alone degrades both. Quotes handle localization; reasons handle correction.
- The most human detail: probability mass withdrawn from the student's errors flows toward self-correction tokens—"Wait," "But," "Actually" appear 2–3× more often under critique conditioning (23.9% vs 4.9% on K&K). The distilled signal encodes a behavioral tendency: the student learns to brake and re-derive.
- Test-time correction: on Olmo-3-32B-Think, failed rollouts get a critique; the chain is spliced at the quoted location, a "Wait, I should correct this reasoning." plus the critique is inserted, and the model continues. For 19 stubborn problems where all 8 samples failed, solve rate rose from 4.3% to 9.3%—doubled, statistically significant. These are the all-wrong group made real.
- Small-scale training (Appendix A): 320 K&K problems, 100 steps—critique-conditioned distillation beats GRPO and reference-conditioned distillation; self-critique recovers most of the gain.
- Cost: self-critiquing adds only 6% step time; an API critic about 20%—recycling compute GRPO wastes on all-wrong groups anyway.
- Training evidence is small-scale (320 problems, 100 steps), not validated in full-scale RL pipelines.
- Critique generation depends on the reference answer—the results show critiques convert privileged information into usable form, not that models can self-rescue without answers.
- Tasks limited to K&K, DeepMath, and ProcessBench.
- Self-critique localization of 18–21% shows small models' self-diagnosis is still weak; noise at scale remains open.
- Maniparambil, M., Rai, A., Verma, E., O'Connor, N., & Reid, F. (2026). A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning. arXiv:2610.13744. NeurIPS 2026 FLLMPT Workshop (oral).
- Zhao et al. 2026a (OPSD); Yang et al. 2026 (RLSD); Pan et al. 2026 (RLCSD); Shao et al. 2024 (GRPO/DeepSeekMath); Yue et al. 2025.
Results are a disturbing report card. With the reference answer as conditioning, the scoring teacher barely reacts at error positions. The error-span Z-score under reference conditioning on Knights & Knaves (K&K) is -0.17—negative, meaning the signal at error locations is *slightly below average*. Across L5-7 and L7-10 difficulty tiers, no setting passes both localization and correction simultaneously. Swapping to an 8B teacher scoring a 1.7B student changes nothing. Worse: using the trajectory-average signal to classify correct vs. incorrect rollouts yields AUC confidence intervals containing 0.5 in all 12 settings—chance level.
The argument is conceptual, not engineering: locating an error is itself a search problem—you must walk the chain, test each step, find the collapsing pivot. Scoring is never required to perform that search. Knowing where the destination is doesn't mean knowing which intersection you turned wrong at.
Finding 2: Same Model, Different Action, Errors Found
The paper's most elegant experiment isolates one variable: the same model, the same rollout, the same reference answer—only the output action changes, from scoring to generating a critique (quote the erroneous sentence, then explain the reason).
Since critic and teacher see identical information, the difference is purely the action. Search is the missing computation. Scoring is a silent computation never forced to run the comparison to completion; generating a critique is explicit—verbatim quotes and step-by-step reasons—forcing the localization reasoning to actually happen.
Finding 3: Critique-Conditioned Distillation Brings the Signal to Life
The fix: generate a critique first, then have the scoring teacher score the rollout conditioned on the critique (with critiques of *other* problems as a placebo control).
Applications
Limitations (As Stated by the Authors)
Closing: Answers Are for People; Critiques Are for Learning
The paper overturns a deep-rooted assumption: that "seeing the answer" equals "knowing where the mistake is." Between the two lies an entire search. Privileged information becomes a usable teaching signal only when organized as "location + reason." When AI gets stuck—the all-wrong group is precisely the mathematical form of being stuck—the answer is not to hand it the answer key. Give it a critique: point out where it went wrong, and explain why.
Answers are for people to read; critiques are what learning consumes. And the 23.9% vs 4.9% "Wait" may be the paper's warmest footnote: good teaching isn't never erring—it's learning to stop right where you're about to.