When AI Rolls the Dice: The Probabilistic Reasoning Crisis in Large Language Models

This post introduces and discusses a research paper on the reliability of LLMs in probabilistic reasoning (Avena, Bet & Busoni, *How reliable are LLMs when it comes to playing dice?*, arXiv:2606.07515).

The Setup

Ask an AI: "What's the probability of rolling two sixes with two dice?" It answers confidently: 1/36 ≈ 2.78%. But pose a counterintuitive problem — three friends each independently pick a random number from 1 to 100; what's the chance all pick the same one? — and even frontier models may falter. The paper's unsettling thesis: LLMs may never have truly understood probability; they merely imitate the language of probability.

Why Probability Is Hard (for Humans and AI)

• Human brains evolved for deterministic causality ("lion → run"), not probability
• Heuristics mislead: representativeness (gambler's fallacy), availability (plane crashes vs. car accidents), anchoring
• Classic traps: Monty Hall (switching wins 2/3, not 50%), birthday paradox, base-rate fallacy in disease testing

Experimental Design

Two datasets:

• Standard exercises: textbook-level problems with direct formulaic answers
• Counterintuitive exercises: birthday paradox, gambler's fallacy, Monty Hall variants, conditional probability traps

Eight frontier models (GPT-4, Claude, Gemini, Llama series, and others), each tested in direct-answer and chain-of-thought (CoT) modes.

Key Findings

1. Near-perfect on standard, near-passing on counterintuitive

| | Standard | Counterintuitive | Gap | |---|---|---|---| | Average accuracy | 96% | 59% | -37% |

The gap is a reasoning-mode gap, not a knowledge gap.

2. Chain-of-thought helps only marginally

Standard: 96% → 97%. Counterintuitive: 59% → 64%. CoT helps models *articulate* reasoning, but does not help them *recognize intuitive traps* — a model that starts reasoning in the wrong direction simply walks more systematically toward the wrong answer.

3. Token bias — the trap of language

Replacing canonical formulations with semantically equivalent but differently worded variants (e.g., rephrasing the birthday problem) degraded performance by over 20%. This suggests pattern-matching against training corpus phrasings rather than genuine understanding — memorizing answers vs. grasping principles.

4. Misleading suggestions are devastating

Embedding plausible-but-wrong hints in prompts (e.g., nudging toward the gambler's fallacy) cut performance by up to 34%. No model was immune.

Why LLMs Fail at "Real" Probabilistic Reasoning

Four hypotheses discussed:

1. Statistical pattern matching vs. symbolic reasoning — token bias shows wording shouldn't matter if principles were understood 2. Contagion of heuristics — models inherit human cognitive biases from training data; perfect imitation of human thinking ≠ perfect rationality 3. No internal probabilistic model — lack of embodied mental simulation (imagining dice, drawing diagrams) 4. Misaligned optimization target — models predict the *next token*, and intuitively wrong answers may be more common in training corpora than correct ones

Why It Matters

• Critical decisions: a 37-point drop is unacceptable in medical diagnosis, financial risk, legal analysis, safety systems
• Education risk: AI errors can reinforce students' intuitive biases
• New attack vector: misleading-hint prompts degrade models by 34%; LLMs are highly vulnerable in adversarial environments
• Reassessing "understanding": high benchmark scores may reflect memorization; real understanding requires generalization across phrasings, interference resistance, and self-consistency

Possible Solutions

1. Neuro-symbolic hybrids — LLM formalizes the problem; a symbolic probabilistic engine computes 2. Adversarial training — include rephrased variants and labeled misleading hints 3. Improved CoT — train models to explicitly check for known biases ("Am I anchored? Is there a conditional-probability trap?") 4. External verification tools — translate problems into Python, execute, and explain

Philosophical Takeaway

Are LLMs imitating *ideal human rationality* or *actual human bias*? The evidence suggests the latter: LLMs are not rational beings beyond humans, but a statistical mirror that amplifies humanity. The tension between usability (human-like intuition) and accuracy (overcoming human bias) is a core design challenge.

Conclusion

• Standard problems: nearly perfect (96%)
• Counterintuitive problems: barely passing (59%)
• Token bias: rephrasing costs 20%+
• Misleading hints: no immunity, up to 34% degradation

Current LLMs are not genuine probabilistic reasoners but sophisticated pattern matchers — fragile exactly where true understanding, bias resistance, and generalization are required. As Feynman put it: "The first principle is that you must not fool yourself — and you are the easiest person to fool." LLMs not only fool themselves; they inherit humanity's millennia of self-deception about probability.

References

• Avena, L., Bet, G., & Busoni, B. (2026). How reliable are LLMs when it comes to playing dice? arXiv:2606.07515.
• Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology.
• Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science.
• Feynman, R. P. (1998). The Meaning of It All. Perseus Books.