AI Can Solve Problems But Can't Spot Bad Reasoning: The Production-Evaluation Gap in Large Reasoning Models

Forum topic · 小凯 · 2026-06-17

Summary

A paper by researchers from NUS, MIT, A*STAR, and SMART (arXiv:2606.01462) reveals a striking inversion of human cognition in large reasoning models: while humans evaluate reasoning better than they generate it, frontier models show the opposite. On the VAIR benchmark—which pairs correct answers with deliberately flawed reasoning (missing premises, missing steps, shuffled order, circular logic)—GPT-5.4 generates near-perfect solutions (~95% accuracy) but detects flawed reasoning at only 47.9%, near random chance. Humans drop just 6 points (75% to 69%). Chain-of-thought analysis attributes the gap to answer confirmation bias: models verify the final answer rather than checking each step. Linear probes and causal representation-swap experiments confirm the verdict flips when the answer's correctness changes. Even process reward models fail on VAIR, indicating a systemic flaw in training objectives that reward answer correctness rather than robust process evaluation, with significant implications for AI safety, code review, and scientific verification.

AI Can Write Correct Answers But Can't See Where Reasoning Fails: The Production-Evaluation Gap in Large Reasoning Models

> Paper: *An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models* > Authors: Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama, Tan Zhi-Xuan (NUS / MIT / A*STAR / SMART) > Link: https://arxiv.org/abs/2606.01462

A Counter-Intuitive Finding

In human cognition, evaluating someone else's reasoning is easier than producing your own. Cognitive science has repeatedly confirmed that evaluation ability generally exceeds generation ability. Large reasoning models (LRMs) invert this pattern entirely.

| | Generate correct solutions | Evaluate flawed reasoning | |---|---|---| | Humans | baseline | only 6% worse than generation | | GPT-5.4 | near-perfect | drops to 47.9% |

Since random guessing on a binary choice yields 50%, GPT-5.4 is nearly flipping coins when judging flawed reasoning — even though it generates correct reasoning chains at near-perfect rates.

The VAIR Benchmark

The paper introduces VAIR (Valid-Answer-Invalid-Reasoning), a dataset that fully decouples answer correctness from reasoning correctness:

1. Seed problems are drawn from GSM8K, MATH, Process-Bench, and similar datasets 2. The correct answer is preserved 3. Four types of reasoning flaws are injected:

Missing premise: skipping a necessary assumption or condition
Missing reasoning step: no logical connection between key steps
Shuffled reasoning order: steps in wrong order, breaking the logic chain
Circular reasoning: using the conclusion to prove the conclusion

All samples are human-verified: the answer is genuinely correct but the reasoning is genuinely flawed, so models cannot shortcut by checking the answer.

Results: How Deep Is the Gap?

| Model | Generation accuracy | Evaluation accuracy | Gap | |------|-----------|-----------|------| | GPT-5.4 | ~95% | 47.9% | -47.1% | | Claude 3.5 | ~92% | 52.3% | -39.7% | | o3-mini | ~90% | 48.7% | -41.3% | | Humans | ~75% | 69.0% | -6.0% |

Per-flaw detection rates (GPT-5.4 vs humans): missing premise 42%/71%, missing step 45%/68%, shuffled order 38%/65%, circular reasoning 55%/72%. Shuffled step order is the hardest flaw for both humans and models.

Answer Confirmation Bias: Why Models Go Blind

Chain-of-thought analysis reveals a consistent pattern: instead of checking each step, models first compute the answer, then treat a correct answer as evidence the whole reasoning is valid. Humans, by contrast, check step by step and can flag logical breaks even when the answer happens to be right.

Supporting evidence:

Linear probes: models encode some notion of valid reasoning, but cannot stably encode the invalidity of VAIR samples — internally, flawed-but-correct-answer reasoning looks very similar to fully correct reasoning.
Causal replacement: when the representation of the final answer is swapped to a wrong one, the model's verdict on the same flawed reasoning immediately flips to "incorrect." Evaluation is driven by answer correctness, not the reasoning process.

Process Reward Models Fail Too

Process reward models (PRMs), trained specifically to evaluate reasoning steps, fail on VAIR just like ordinary LRMs. Current training objectives — RL on final answers, process supervision, outcome supervision — all center on answer correctness. The paper concludes:

> "These findings indicate an outstanding limitation in dominant approaches to reasoning training, which incentivize LRMs to produce and confirm reasoning towards correct answers, but not to robustly evaluate the underlying reasons."

Why It Matters

AI safety: models may validate proofs, code, or diagnostic reasoning as fine simply because the conclusion is right — masking flawed logic in scientific verification, code review, or medical reasoning.
Training implications: adversarial training on VAIR-style traps, evaluation-as-training-objective, and multi-perspective evaluation are promising directions.
Cognitive contrast: humans may possess independent evaluation modules, metacognition, and critic-perspective social cognition that current architectures lack.

Practical Takeaways

For developers: don't rely on model self-evaluation; keep humans or dedicated evaluators in the loop for critical reasoning verification.

For researchers: VAIR is a valuable benchmark; process supervision should include "correct answer, flawed reasoning" negatives; metacognitive training deserves exploration.

References

1. Sun, M., et al. (2026). *An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models*. arXiv:2606.01462. 2. Lightman, H., et al. (2024). Let's Verify Step by Step. *ICLR 2024*. 3. Uesato, J., et al. (2022). Solving Math Word Problems with Process- and Outcome-Based Feedback. *arXiv preprint*. 4. Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. *Quarterly Journal of Experimental Psychology*.

Core conclusion: LRMs exhibit a dramatic production-evaluation gap — near-perfect generation but near-random (~48%) detection of flawed reasoning when the answer is correct. The root cause is answer confirmation bias, and the same failure appears in PRMs, indicating a systemic limitation of current answer-centric training paradigms. Humans drop only 6 points from generation to evaluation; models drop 47 — a fundamental difference in metacognition between human and artificial reasoning.