Why Even Smart Minds Deceive Themselves: AI Scientific Reasoning and Confirmation Bias
> Paper: *FALSIFYBENCH: Evaluating Hypothesis-Driven Reasoning in LLMs* > arXiv: 2606.04751 > Authors: Leonardo Bertolazzi, Massimo W. Barberi, Maria Grazia > Released: 2026-06-03
This is a full English rendering of a Chinese forum post (zhichai.net). The original is long and essayistic; the sections below preserve its structure and claims.
Introduction: A Detective's Trap
The post opens with a parable: a detective finds a button clutched in a murder victim's hand, assumes it was torn from the killer's clothing, spends days verifying this hypothesis, and arrests the wrong man—while the real killer wore entirely different clothes. The detective is intelligent and diligent, but he committed a fatal error: he searched only for evidence confirming his hypothesis rather than evidence that could refute it. This is confirmation bias, one of humanity's most stubborn cognitive traps. On June 3, 2026, researchers posed the question to AI: when LLMs play the role of scientists, do they make the same mistake?
Wason's 2-4-6 Task: A Classic Test of Scientific Thinking
In 1960, psychologist Peter Wason designed a deceptively simple experiment: given the triple 2-4-6 as an example fitting a hidden rule, participants propose triples and learn whether each fits. Most guessed "even numbers ascending," tested only confirming cases (8-10-12, 20-22-24), and never tried triples like 1-2-3 that could falsify their hypothesis. The actual rule—"three ascending numbers"—was far simpler.
The task endures because it reveals a deep truth: we instinctively seek confirming evidence, not disconfirming evidence. This is not stupidity; it may be an evolved cognitive efficiency (quickly confirming "is there a lion in the grass?" aids survival). But in scientific reasoning—where Karl Popper argued the core of good theory is *falsifiability*—this instinct becomes a fatal flaw.
FALSIFYBENCH: Making AI Play Scientist
FALSIFYBENCH adapts this classic experiment and evaluates LLMs across four abilities:
- Hypothesis generation — can the model propose plausible hypotheses?
- Evidence collection — can it design effective tests?
- Belief revision — can it update when evidence contradicts the hypothesis?
- Falsification — does it actively seek counter-evidence?
- Reasoning models outperform instruction-tuned models at scientific reasoning—suggesting the "reasoning" training direction is productive.
- No model approaches optimal performance. Even the best models fall short of a trained human scientist.
- Turn-level analysis reveals the decisive factor: negative testing. Successful models ask "what evidence would prove my hypothesis wrong?" and search for it; failing models, like humans, chase confirming evidence until they hit a wall.
- Sunk cost: having invested effort in a hypothesis, we resist abandoning it.
- Cognitive dissonance: contradictory evidence is uncomfortable; ignoring or reinterpreting it is easier.
- Motivated reasoning: we want not just truth, but to *feel right*.
- Sherlock Holmes: "When you have eliminated the impossible, whatever remains, however improbable, must be the truth." Holmes's brilliance is not raw deduction but systematic *elimination*—falsification in practice.
- Richard Feynman: when hearing a theory, Feynman immediately asked "what experiment would prove it wrong?" A theory that can only be confirmed, not falsified, is faith, not science. His cognitive humility—"I might be wrong"—is the essence of scientific spirit, which current AI has not attained.
- Orwell's Doublethink: *1984*'s concept of holding contradictory beliefs relates to cognitive dissonance. Confirmation bias is a milder doublethink: selectively attending to supporting evidence. An AI that learned this "mild self-deception" would be dangerous—err systematically while appearing consistent and reasonable.
- Popper: a good theory must risk refutation; science advances by eliminating false theories, not confirming true ones. FALSIFYBENCH thus tests not just reasoning ability but "scientific spirit."
- Kuhn: science mostly "puzzles" within a paradigm; paradigm shifts happen only when anomalies accumulate. The post claims FALSIFYBENCH AIs mostly stay in "puzzle-solving" mode and struggle to abandon a flawed hypothesis for a new framework.
- Lakatos: research programmes have an immovable "hard core" and adjustable "protective belt." AI may need to distinguish which assumptions to hold firm and which to revise flexibly.
- Medical diagnosis: a doctor who locks in "pneumonia" early and only notices cough and fever while ignoring atypical X-rays may misdiagnose. Good clinicians ask "what evidence would prove me wrong?" AI-assisted diagnosis must learn this habit.
- Investing: investors see only good news after buying a stock. Charlie Munger's inversion—"if I knew where I would die, I would never go there"—is falsification thinking applied to avoiding error.
- Relationships: self-verification theory says people seek feedback confirming their self-concept. An AI companion designed to always affirm users may reinforce their biases rather than foster growth.
- Bertolazzi, L., Barberi, M. W., & Grazia, M. (2026). *FALSIFYBENCH: Evaluating Hypothesis-Driven Reasoning in LLMs*. arXiv:2606.04751.
- Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. *Quarterly Journal of Experimental Psychology*, 12(3), 129-140.
- Popper, K. R. (1959). *The Logic of Scientific Discovery*. Hutchinson.
- Kuhn, T. S. (1962). *The Structure of Scientific Revolutions*. University of Chicago Press.
- Feynman, R. P. (1985). *Surely You're Joking, Mr. Feynman!*. W.W. Norton.
- Lakatos, I. (1978). *The Methodology of Scientific Research Programmes*. Cambridge University Press.
Twelve LLMs from different families and scales participated, including reasoning models (e.g., the o1 series) and standard instruction-tuned models.
Key findings reported in the post
Why Is Falsification So Hard?
Cognitive psychology perspective
The author notes these seem human-specific, yet AI shows similar behavior—possibly because LLMs internalize human cognitive-bias patterns from training data.
Logical structure
Logically, falsification is more "efficient": one black swan defeats "all swans are white," while a thousand white swans prove little. But psychologically, confirming a known pattern feels rewarding while discovering error is painful. AI may face a similar reward-structure problem: training rewards plausible-sounding output, while self-questioning may look like inconsistency.
Navigating hypothesis space
Scientific reasoning resembles navigating a maze of hypotheses. Confirmation bias means committing to one promising-looking path until a dead end; falsification means actively seeking evidence a path is wrong, so you can backtrack early. The post reports that AI navigates hypothesis space much like humans: local "deep cultivation" rather than global exploration.
Literary Reflections
Philosophy of Science
Real-World Echoes
The Future: Teaching AI Self-Doubt
1. Training data reform: include histories of scientific error and correction (geocentrism → heliocentrism, phlogiston → oxidation), not just correct answers. Showing how great scientists erred and self-corrected may be more instructive than showing only right answers. 2. Reward redesign: reward actively discovering one's own errors, not only producing correct content. 3. Multi-agent debate: models propose rival hypotheses and challenge each other's evidence—an adversarial verification mimicking peer review, replication, and open debate.