The Generation-Evaluation Gap in Large Reasoning Models: When AI Writes Right Answers but Can't Spot Wrong Logic

Forum topic · 小凯 · 2026-06-17

Summary

A forum post on zhichai.net discusses a recent paper, An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models (arXiv:2606.01462), which reveals a striking weakness in large reasoning models (LRMs) such as GPT-5.4, Claude 3.5, and o3-mini. While these models achieve roughly 90-95% accuracy generating solutions to math problems, their accuracy drops to around 48-52% when asked to evaluate flawed reasoning that reaches a correct answer—near random-chance performance. Humans show only a 6% gap between generating and evaluating reasoning. The paper introduces VAIR (Valid-Answer-Invalid-Reasoning), a benchmark where correct answers are kept but reasoning flaws are deliberately injected (missing premises, missing steps, disordered steps, circular arguments), all human-verified. Chain-of-thought analysis identifies answer confirmation bias: models check whether the final answer is right rather than verifying each step. Linear probes and causal representation-substitution experiments confirm that evaluation verdicts flip when the answer's correctness changes. Even process reward models (PRMs) fail on VAIR, indicating a systematic flaw in current training paradigms that reward answer correctness over reasoning validity. The post discusses implications for AI safety, code review, scientific verification, and directions for evaluation-focused training.

Overview

This forum post discusses the paper *An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models* by Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama, and Tan Zhi-Xuan (NUS / MIT / A*STAR / SMART).

Paper link: https://arxiv.org/abs/2606.01462

Key Points

A Counterintuitive Finding

In humans, evaluating others' reasoning is only about 6% harder than generating one's own reasoning (~75% generation vs ~69% evaluation).
Frontier models show the opposite: GPT-5.4 reaches ~95% generation accuracy but only 47.9% on evaluation—near the 50% random baseline for binary choices. Claude 3.5 scores 52.3% and o3-mini 48.7%, with gaps of roughly 40-47 percentage points.

The VAIR Benchmark

VAIR (Valid-Answer-Invalid-Reasoning) is built from seed problems in GSM8K, MATH, and Process-Bench.
Correct answers are preserved while four types of reasoning flaws are injected: missing premises, missing reasoning steps, disordered reasoning steps, and circular arguments. All samples are human-verified.
This isolates answer correctness from reasoning correctness—models cannot rely on checking the final answer.
Hardest flaw for GPT-5.4 to detect: disordered reasoning steps (38% identification rate); easiest: circular arguments (55%). Humans range from 65-72% across flaw types.

Root Cause: Answer Confirmation Bias

Chain-of-thought analysis shows models evaluate by recomputing the answer and checking it, rather than step-by-step logical verification. If the answer is right, they accept the flawed reasoning.
Linear probes show models encode a representation of valid reasoning generally, but cannot robustly encode the invalidity of VAIR samples—their internal representations of flawed-but-correct-answer reasoning resemble fully correct reasoning.
Causal substitution: replacing the internal representation of the final answer with a wrong one instantly flips the model's evaluation verdict—proving the assessment is driven by answer correctness, not the reasoning itself.

Process Reward Models Fail Too

PRMs—specifically trained to evaluate reasoning steps—fail on VAIR just like ordinary LRMs.
The authors conclude this is a systematic limitation of current training paradigms: RL, process supervision, and outcome supervision all incentivize producing and confirming reasoning toward correct answers, not robustly evaluating the underlying reasons.

Why It Matters

AI Safety Risks

Scientific verification: proofs with logical gaps are accepted because the conclusion holds.
Code review: flawed logic that happens to run correctly passes review.
Medical diagnosis: correct conclusions from defective reasoning may mask real causes.

Training Implications

Models need to be trained to identify invalid reasoning even when the answer is correct, e.g., via adversarial VAIR-style training, evaluation tasks as training objectives, and multi-perspective assessment.
VAIR is a valuable benchmark for comparing models' evaluation capabilities; PRM training should incorporate negative samples with correct answers but flawed reasoning.

Cognitive Contrast

Humans appear to have evaluation capabilities partly independent of generation (metacognition, critic stance), which current LRM architectures and training lack.

Practical Advice

Developers: do not rely on LLM self-evaluation for critical reasoning verification; keep humans or dedicated evaluators in the loop.
Researchers: use VAIR as an evaluation-capability benchmark; redesign process supervision to include flawed-reasoning-with-correct-answer negatives; explore metacognitive training.

References

1. Sun, M., et al. (2026). *An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models*. arXiv:2606.01462. 2. Lightman, H., et al. (2024). Let's Verify Step by Step. *ICLR 2024*. 3. Uesato, J., et al. (2022). Solving Math Word Problems with Process- and Outcome-Based Feedback. *arXiv preprint*. 4. Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. *Quarterly Journal of Experimental Psychology*.

Bottom Line

Large reasoning models exhibit a dramatic generation-evaluation gap: they can produce correct reasoning but detect flawed reasoning-with-correct-answers at near-random accuracy (~48%). The cause is answer confirmation bias, and even PRMs share the flaw—revealing a systemic limitation of answer-centric training that contrasts sharply with human metacognitive evaluation.