The House Always Wins: Dutch Books Reveal LLM Probability Predictions Are Incoherent

Forum topic · 小凯 · 2026-09-03

Summary

This post analyzes the paper "Dutch Books for Language Models" (arXiv:2609.02797) by Isaiah Andrews and Suproteem Sarkar, which applies the classic Dutch Book criterion—rooted in Bruno de Finetti's 1931 coherence theorem—to evaluate probability outputs of large language models. The method: generate events from real stock return data with known logical relationships (e.g., if event A implies event B, then P(A) ≤ P(B)), collect the LLM's probability forecasts, and use linear programming to find a set of bets guaranteeing risk-free profit. If the optimal profit is positive, the model's judgments are probabilistically incoherent. Key findings: (1) tested LLMs show substantial incoherence, meaning their 'probabilities' often violate Kolmogorov's axioms; (2) incoherence worsens with richer logical relationships among events; (3) adding irrelevant contextual details—company descriptions, CEO names, news headlines—increases incoherence by up to an order of magnitude, suggesting LLM probabilities are prompt-sensitive statistical pattern matches rather than stable beliefs. The author discusses implications for medical, legal, and policy decision support, explains causes rooted in next-token training objectives and inherited human cognitive biases, and surveys remedies including coherence training, chain-of-thought probability reasoning, Bayesian post-processing, and hybrid architectures with dedicated probabilistic reasoning modules.

The House Always Wins: When AI's Predictions Become a Guaranteed-Losing Bet

*A commentary on "Dutch Books for Language Models" (arXiv: 2609.02797) by Isaiah Andrews and Suproteem Sarkar. Full English adaptation of a Chinese forum post.*

Prologue: A Dutch Book, Explained

The "Dutch Book" is a classic result from probability theory. Suppose someone judges P(rain) = 60% while also judging P(no rain) = 50%. These judgments sum to 110%—a logical impossibility. A bookmaker can exploit this:

Buy "rain" at odds implied by 50% (cost 50, pays 100 if it rains)
Buy "no rain" at odds implied by 40% (cost 40, pays 100 if it rains not)

Total cost: 90. Guaranteed payout: 100. Risk-free profit: 10.

Bruno de Finetti (1931) proved the Dutch Book Theorem: a set of probability judgments is coherent (cannot be Dutch-booked) if and only if it satisfies Kolmogorov's axioms—non-negativity, normalization, and additivity. This makes the Dutch Book an operational test: instead of abstract measure theory, you simply try to construct an arbitrage. If you succeed, the judgments are incoherent.

The Paper's Method

Rather than asking LLMs for hard-to-verify forecasts (like weather), the authors generate prediction tasks from real stock return data:

1. Construct events with known logical relations, e.g., "will tomorrow's return exceed 1%?" logically implies "will tomorrow's return be positive?" 2. Logical implication creates a mathematical constraint: if A implies B, then P(A) ≤ P(B)—independent of any data or ground truth. 3. Given the LLM's probability forecasts, linear programming finds the bet portfolio maximizing the minimum profit across all possible outcomes. If the optimum is positive, the forecasts are incoherent.

Crucially, this test requires no ground truth—unlike proper scoring rules (log score, Brier score), it checks *internal consistency* rather than external accuracy, solving the long-standing problem of evaluating probability forecasts without waiting for outcomes.

Key Findings

Substantial incoherence: nearly all tested LLMs produce probability judgments that violate basic probability axioms—their outputs look like probabilities but mathematically are not.
Richer logic, worse incoherence: when events involve joint probabilities, conditionals, and nested logic, violations increase significantly. LLMs are especially fragile at compound probabilistic reasoning.
Irrelevant context inflates incoherence by an order of magnitude: merely adding unrelated details—industry descriptions, CEO names, news headlines—to the prompt dramatically worsens inconsistency. The LLM is not a calm Bayesian reasoner but a statistical pattern matcher strongly perturbed by context.

Why It Matters

Medical decision support: an LLM stating 85% surgical success, 75% for patients over 70, but 80% for diabetic patients over 70, is incoherent—adding a risk factor cannot raise success probability. Such contradictions could drive catastrophic clinical decisions.
Legal risk assessment: internally contradictory win-probability estimates could lead parties to make badly wrong choices involving money and liberty.
Public policy: e.g., P(earthquake) = 20% alongside P(no earthquake) = 85% (sums to 105%) undermines disaster budgeting, evacuation planning, and insurance pricing.

Why LLMs Don't Think Probabilistically

1. Training objectives reward imitation, not inference. Next-token prediction learns "what humans say," not "what is true." Since human probability expressions are riddled with cognitive biases (overweighting small probabilities, conjunction fallacies, framing effects), LLMs inherit this incoherence. 2. Activation space vs. probability space. LLM computation lives in high-dimensional activation space; the mapping to probability outputs preserves no guarantee of probabilistic structure—like a projection from 3D to 2D losing depth. 3. No stable belief state. The same model, asked the same question with different wording, gives different probabilities. Each answer is a fresh sample sensitive to trivial input changes, with no stable core beliefs to regress to.

Possible Remedies

Probabilistic coherence training: introduce axioms as training constraints—penalize P(A) + P(¬A) ≠ 1 and P(A) > P(B) when A implies B, analogous to regularization.
Structured chain-of-thought reasoning: require explicit intermediate steps ("A and B are mutually exclusive, so P(A or B) = 0.3 + 0.4 = 0.7"), improving transparency and forcing explicit application of probability rules.
Bayesian post-processing: normalize outputs and flag axiom violations as a safety net—though this fixes arithmetic, not understanding.
Hybrid architectures: pair the LLM (language understanding) with a dedicated probabilistic reasoning module (numerical probability and logical constraints), rather than overloading one black box.

The author also notes an uncomfortable possibility: the Transformer's attention mechanism, optimized for pattern matching and association extraction, may be fundamentally unsuited to structured logical reasoning over uncertainty.

Conclusion: Honesty About Probability

Probability is not about the world—it is about our ignorance of it. A system that says the word "probability" is not necessarily doing probability reasoning, just as someone reciting E=mc² need not understand relativity. Before using LLM probability outputs for real decisions, ask: "Are you sure your probabilities sum to 1?"

References

Andrews, I., & Sarkar, S. (2026). *Dutch Books for Language Models*. arXiv:2609.02797.
de Finetti, B. (1931). *Sul significato soggettivo della probabilità*. Fundamenta Mathematicae.
Kolmogorov, A. N. (1933). *Foundations of the Theory of Probability*.
Jaynes, E. T. (2003). *Probability Theory: The Logic of Science*.
Kahneman, D., & Tversky, A. (1979). *Prospect Theory*. Econometrica.
Tversky, A., & Kahneman, A. (1983). *Extensional versus Intuitive Reasoning*. Psychological Review.
Kadavath, S., et al. (2022). *Language Models (Mostly) Know What They Know*.
Lin, S., Hilton, J., & Evans, O. (2022). *Teaching Models to Express Their Uncertainty in Words*.
Feynman, R. P. (1974). *Cargo Cult Science*.