When AI Rolls the Dice: The Probabilistic Reasoning Crisis in Large Language Models

🎲 Introduction: A Philosophical Question About Dice

Imagine asking an AI: "What is the probability of rolling two sixes with two dice?"

If it answers "1/36, about 2.78%"—you nod, satisfied.

But if you ask: "Three friends each independently pick a random number from 1 to 100. What is the probability they all pick the same number?"—it may hesitate, or give an intuitively plausible but mathematically wrong answer.

This is not because AI cannot calculate. On the contrary, today's most advanced LLMs perform remarkably on advanced mathematics—solving calculus, proving theorems, writing code. But when facing counterintuitive probability problems, they behave just like humans—misled by intuition and blinded by biases.

This paper reveals an unsettling truth: LLMs may never truly understand probability—they merely imitate the language of probability.

🧠 Probabilistic Reasoning: A Shared Weakness of Humans and AI

The Monty Hall Problem: An Intuition Trap

The classic paradox: three doors hide a car and two goats. After you pick a door, the host (who knows what's behind them) opens another door revealing a goat, then asks: "Do you want to switch?"

Intuition says switching doesn't matter—50/50.

Mathematics says switching wins 2/3 of the time; staying wins only 1/3.

Most humans get it wrong. Surprisingly, many LLMs get it wrong too—or get it right with flawed reasoning.

Why Is Probability So Counterintuitive?

• Our brains evolved for deterministic causality ("lion approaches → run"), not probability
• Mental heuristics aid survival but often fail mathematically
• Representativeness heuristic: assuming "random" means "evenly distributed" (gambler's fallacy)
• Availability heuristic: easily recalled examples seem more likely (plane crashes vs. car accidents)
• Anchoring effect: the first number seen influences subsequent judgment

🧪 Experimental Design: Making AI Play Dice

The researchers built two datasets to systematically test LLM probabilistic reasoning:

Dataset 1: Standard Exercises

Textbook-level problems, such as:

• "Rolling a fair die, what is the probability of an even number?"
• "Drawing one card from 52, what is the probability it's a heart?"
• "For independent events A and B with P(A)=0.3, P(B)=0.4, what is the probability both occur?"

Characteristics: direct, explicit, formula-conformant.

Dataset 2: Counterintuitive Exercises

Problems designed to trigger heuristic thinking, such as:

• Birthday paradox: 23 people give >50% chance of a shared birthday
• Gambler's fallacy: after 5 heads in a row, what is the probability the 6th flip is heads? (Still 50%)
• Variants of the Monty Hall problem
• Conditional probability traps: a test with 95% accuracy and 1% disease prevalence—what is the actual probability of disease given a positive result? (Not 95%)

Characteristics: answers violate intuition; strict conditional probability computation is required.

Models Tested

Eight state-of-the-art models, including GPT-4, Claude, Gemini, and Llama series, plus other open-source models—each tested in two modes:

• Direct answering
• Chain-of-Thought (CoT)

📊 Results: The Huge Gap Between Standard and Counterintuitive Problems

Finding 1: Near-Perfect on Standard, Barely Passing on Counterintuitive

| Model Type | Standard Accuracy | Counterintuitive Accuracy | Gap | |---------|------------|-------------|------| | Average | 96% | 59% | -37% |

This means:

• LLMs are nearly perfect at direct calculation
• On problems requiring overcoming intuition, performance barely exceeds random guessing (for binary choices)
• The gap is not a knowledge gap but a reasoning-mode gap

Finding 2: Chain-of-Thought Helps Only Marginally

• Standard + CoT: 96% → 97% (nearly no gain)
• Counterintuitive + CoT: 59% → 64% (small gain, still poor)

Key insight: CoT helps models express reasoning more carefully, but does not help them recognize intuition traps. If the model starts down the wrong path, CoT just makes it systematically wrong.

Finding 3: Token Bias—The Trap of Language

The most shocking finding.

Design: replace canonical problem formulations with semantically equivalent but differently worded "disguised variants"—e.g., "at least two people share a birthday" becomes "there exists a pair of people with the same birthday."

Result:

• Performance drops by more than 20%
• LLMs are biased toward specific words, phrasings, and expressions
• They are not truly understanding the problem—they are matching patterns from training corpora

Deeper implication:

• LLMs may have seen the canonical "birthday paradox" formulation and thus "know" the answer
• When the problem is reworded, they fail to transfer knowledge
• This resembles memorizing answers vs. understanding principles

Finding 4: The Devastating Effect of Misleading Suggestions

Design: embed "plausible but mathematically wrong" hints in prompts—e.g., "Hint: many believe previous outcomes should be considered here" (implying the gambler's fallacy).

Result:

• Performance drops by up to 34%
• No model is immune
• Even the best models are significantly swayed

Deeper implication:

• LLMs are highly sensitive to social cues and authoritative suggestions
• Similar to human conformity, but harmful in mathematical contexts
• A user (or malicious attacker) supplying wrong hints can easily derail the model

🔍 Deep Analysis: Why LLMs Don't "Truly" Reason About Probability

Hypothesis 1: Statistical Pattern Matching vs. Symbolic Reasoning

LLM probabilistic "reasoning" is really sophisticated pattern matching, not genuine symbolic computation. Evidence: token bias experiments show wording changes cause large performance drops—impossible if the model truly understood the principles. Like a student who memorized the birthday paradox solution without grasping combinatorics.

Hypothesis 2: The "Contagiousness" of Heuristic Thinking

LLMs learn common human probabilistic misconceptions from training data, thus inheriting human cognitive biases. They perform well on "typical" distributions and poorly on "atypical" ones—suggesting they learned the statistical regularities of human intuition, not mathematical truth. Perfectly imitating human thinking ≠ perfectly rational thinking.

Hypothesis 3: Lack of an Internal Probabilistic Model

Humans can aid reasoning through internal simulation (imagining dice rolling) or sketching diagrams. LLMs, especially in text-only mode, lack this embodied simulation ability—geometric and spatial probability problems are particularly hard for them.

Hypothesis 4: Misaligned Optimization Objective

LLMs are trained to predict the next token, not to reason correctly. In training corpora, intuitive wrong answers may be more common than correct mathematical ones—so the model optimizes for "predicting human intuition" rather than "predicting mathematical truth."

⚠️ Why This Matters

1. Reliability in Critical Decisions

If LLMs are used for medical diagnosis (disease/side-effect probabilities), financial decisions (risk assessment), legal analysis (evidence weighting), or safety systems (failure probabilities), a 37% accuracy drop is unacceptable.

2. Risks as Educational Tools

If students learn probability from LLMs: standard problems confirm understanding; counterintuitive failures make students blame the problem—the AI's errors reinforce students' intuitive biases.

3. A New Vector for Adversarial Attacks

Misleading-hint attacks can degrade performance by 34%, with no model immune—LLMs are extremely vulnerable in adversarial environments.

4. Reassessing AI "Understanding"

High math scores may come from memorized training-data answers. Genuine understanding requires generalization to new formulations, robustness against misleading hints, and self-consistency across phrasings.

🔧 Possible Solutions

1. Symbolic-neural hybrid architectures: the LLM parses the problem into a formal probabilistic model; a symbolic engine (probabilistic graphical models, Bayesian networks) computes; the LLM translates results back into natural language.

2. Adversarial training: train on reworded variants and deliberately misleading hints labeled with correct answers, teaching models to recognize and ignore misdirection. Challenge: covering all possible misdirections requires massive samples.

3. Improved chain-of-thought: not just "think step by step" but explicitly checking for biases—"Am I affected by anchoring?" "Is there a conditional probability trap here?"—leveraging self-reflection without extra architecture.

4. External verification tools: have the model translate probability problems into Python code, execute it for exact answers, and pair with natural-language explanations.

🌍 Broader Philosophical Reflections

Whom Are LLMs Imitating?

Are LLMs imitating ideal human rationality or actual human biases? The results suggest the latter: standard problems good, counterintuitive problems bad, swayed by wording and misdirection. LLMs are not superhuman rational agents, but statistical mirrors amplifying humanity.

The Definition of Intelligence

If AI must "think like humans," it will err like humans. If AI must be "perfectly rational," it may become unintelligible, abandoning human intuitive frameworks. This is the core tension of AI design: usability (matching human intuition) vs. accuracy (surpassing human bias).

The Nature of Probability

Probability theory is a human invention for handling uncertainty—yet the human brain did not evolve for it. LLMs learn human language and thought, but probability is one of the least intuitive domains of that language. Perhaps their failures remind us: true intelligence is not imitating humans, but remaining rational in the face of human weakness.

🎯 Conclusion

The paper's title asks a seemingly simple question: "How reliable are LLMs when playing dice?"

The answer is far more complex:

• Nearly perfect on standard problems (96%)
• Barely passing on counterintuitive ones (59%)
• Affected by token bias (rewording → 20% drop)
• No immunity to misleading hints (misdirection → 34% drop)

Core conclusion: current LLMs are not true probabilistic reasoners. They are sophisticated pattern matchers—flawless where training data covers the scenario, fragile where genuine understanding, bias resistance, and generalization are required.

This is not an LLM "failure" but an exposure of our misconceptions about their capabilities. When we treat LLMs as omniscient AI, this paper is a wake-up call—they remain mirrors of human wisdom, not transcendences of human rationality.

Perhaps, as Feynman said: "The first principle is that you must not fool yourself—and you are the easiest person to fool."

The problem with LLMs is that they not only fool themselves, but also inherit humanity's millennia of self-deception about probability.

---

References

• Avena, L., Bet, G., & Busoni, B. (2026). How reliable are LLMs when it comes to playing dice? arXiv:2606.07515.
• Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology.
• Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science.
• Feynman, R. P. (1998). The Meaning of It All. Perseus Books.