Dutch Books for Language Models: When AI Probability Predictions Become a Guaranteed Loss

Forum topic · 小凯 · 2026-09-03

Summary

This post is a Chinese-language commentary on the paper "Dutch Books for Language Models" (arXiv:2609.02797) by Isaiah Andrews and Suproteem Sarkar. The paper applies the classic Dutch Book theorem, originating with Bruno de Finetti, to test whether large language models produce probabilistically coherent predictions. The method generates probability-forecasting tasks from real stock-return data, where events have known logical relationships (e.g., A implies B, so P(A) ≤ P(B)), then uses linear programming to find risk-free arbitrage opportunities—Dutch Books—against the model's stated probabilities. A key advantage is that the test requires no ground-truth outcomes, only internal consistency. Reported findings: LLMs show substantial probabilistic incoherence; incoherence grows with richer logical structure; and irrelevant contextual details (industry descriptions, CEO names, news headlines) can increase incoherence by an order of magnitude. The commentary discusses implications for medical, legal, and public-policy decision support, traces causes to next-token training that inherits human cognitive biases, and outlines remedies such as coherence-aware training, chain-of-thought probability reasoning, and Bayesian post-processing, while acknowledging possible architectural limits.

Introduction: The Dutch Book Concept

A *Dutch Book* is a set of bets that guarantees a profit for one side regardless of outcomes. If someone judges P(rain) = 60% and P(no rain) = 50% (summing to 110%), a bettor can exploit the inconsistency: stake 50 at odds implied by 50% on "no rain" and 40 at odds implied by the complement of 60% on "rain". Total cost: 90; guaranteed payout: 100; risk-free profit: 10.

Bruno de Finetti (1931) proved the Dutch Book Theorem: a set of probability judgments is *coherent* (immune to Dutch Books) if and only if it satisfies Kolmogorov's probability axioms—non-negativity, normalization, and additivity for mutually exclusive events. The paper *Dutch Books for Language Models* (arXiv:2609.02797, Isaiah Andrews & Suproteem Sarkar) applies this test to LLMs.

Methodology

Task generation: Instead of hard-to-verify questions like weather forecasts, the authors build forecasting events from real historical stock-return data, e.g., "Will this stock's return tomorrow exceed 1%?" Events carry logical relationships (A implies B ⇒ P(A) ≤ P(B)).
Linear programming: Bet stakes on each event are variables; logical relationships are constraints; the objective maximizes the minimum profit across all possible outcome realizations. If the optimum is positive, a Dutch Book exists and the model's judgments are incoherent.
No ground truth needed: Unlike proper scoring rules (log score, Brier score), the test evaluates *internal consistency*, not external accuracy—so coherence can be assessed immediately.

Key points

Substantial incoherence: Nearly all tested LLMs produce probability judgments that violate basic axioms—their "probabilities" are often not probabilities in the mathematical sense.
Logical complexity amplifies the problem: Incoherence grows significantly when events involve joint events, conditional structure, and nested logic—LLMs are especially fragile at compound probabilistic reasoning.
Irrelevant context is devastating: Adding contextually unrelated details (industry descriptions, CEO names, recent headlines) to prompts increased incoherence by up to an order of magnitude, suggesting LLM probabilities are prompt-sensitive outputs rather than stable belief states.

Why it matters

Medical decision support: A model may claim 75% success for patients over 70 but 80% for patients over 70 *with diabetes*—an added risk factor raising success probability is a coherence violation with potentially dangerous consequences.
Legal risk assessment: Inconsistent win-probability estimates under different evidence scenarios can drive wrong litigation choices.
Public policy: Complementary event probabilities summing to 105% undermine resource allocation, insurance pricing, and disaster planning built on them.

Root causes

LLMs are trained on next-token prediction—they imitate how humans *talk* about probability, and since human probability judgments are notoriously biased (overweighting small probabilities, conjunction fallacies, framing effects), models inherit these inconsistencies.
Internal computation lives in activation space; the mapping from activations to probability outputs does not preserve the structure required by probability theory.
Probabilities are highly context-dependent: the same question phrased differently can yield different estimates, with no stable core belief to regress to.

Possible remedies

1. Coherence-aware training: penalize outputs where P(A) + P(¬A) ≠ 1 or where P(A) > P(B) despite A ⇒ B, using synthetic data with known logical relations. 2. Structured chain-of-thought probability reasoning: require explicit intermediate steps applying probability rules, improving transparency and verifiability. 3. Bayesian post-processing: a safety-net layer that normalizes probabilities and flags or repairs basic violations—though it fixes arithmetic, not understanding.

The author notes a fundamental caveat: Transformer attention is built for pattern matching rather than structured logical inference, so LLMs may never reason probabilistically natively. A hybrid architecture—LLM for language, a dedicated probabilistic reasoning module for numeric coherence—may be the realistic path forward.

Conclusion

The takeaway: a system that emits the word "probability" does not thereby perform probabilistic reasoning. Before using LLM probability outputs in real decisions, ask first: "Do your probabilities even add up to 1?"

References

Andrews, I., & Sarkar, S. (2026). *Dutch Books for Language Models*. arXiv:2609.02797.
de Finetti, B. (1931). *Sul significato soggettivo della probabilità*. Fundamenta Mathematicae.
Kolmogorov, A. N. (1933). *Foundations of the Theory of Probability*.
Kahneman, D., & Tversky, A. (1979). *Prospect Theory*. Econometrica.
Tversky, A., & Kahneman, D. (1983). *Extensional versus Intuitive Reasoning*. Psychological Review.
Kadavath, S., et al. (2022). *Language Models (Mostly) Know What They Know*.
Lin, S., Hilton, J., & Evans, O. (2022). *Teaching Models to Express Their Uncertainty in Words*.