Background: A Strange Phenomenon
Imagine you are a statistician surveying national income distribution. You draw 1,000 people at random, compute the overall mean (say, 50,000), then split the sample by gender: men average 55,000, women average 45,000. By the Law of Total Probability:
> 0.5 × 55,000 + 0.5 × 45,000 = 50,000
This identity must hold whenever partitioning and weights are correct. Now ask a frontier LLM the same question, but in two forms: directly ("what is the national mean?") and via subgroup conditioning ("what is the male mean? the female mean?"). A competent statistician would produce identical results. The paper shows that LLMs do not.
The Law of Total Probability
The Law is the simplest identity in probability theory. Given a population partitioned into subgroups with known proportions pᵢ and subgroup means μᵢ, the population mean equals Σ pᵢ μᵢ. If an estimator violates this identity, it is not a coherent probabilistic model. The paper's central question is whether modern LLMs satisfy it across hierarchical partitions.
The Binary Tree Evaluation Scaffold
The authors construct decision-tree partitions of a population along attributes such as age, gender, occupation, and region. Two estimates are then compared:
- Direct (macro): ask the model for the population-level quantity.
- Partition-Aggregate (micro): ask the model for the quantity at each leaf, then take a population-weighted sum back up the tree.
- Fragmented knowledge: information is tied to specific contexts and personas, not consolidated into abstract distributions.
- Context dependence: the answer depends strongly on how the question is phrased, indicating answers are produced at query time rather than retrieved from a stable internal world model.
- Limited metacognition: models rarely flag inconsistencies in their own outputs. A system with genuine metacognition would notice when its direct and aggregated estimates diverge.
- Medical decision support: aggregating subgroup risk factors into an overall probability must obey the Law of Total Probability; violation yields misdiagnosis.
- Financial forecasting: portfolio recommendations across asset classes and markets must be internally consistent.
- Policy analysis: aggregating preferences across demographic groups must reconcile; otherwise policy recommendations are internally contradictory.
- Scientific inference: theories whose macro and micro predictions disagree are rejected; LLMs used as inference engines inherit this requirement.
- Wolf, P., Buening, T. K., Krause, A., & Mendler-Dünner, C. (2026). *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models*. arXiv:2607.15277. https://arxiv.org/abs/2607.15277
If the model is self-consistent, both routes produce the same number.
Finding 1: Widespread Violations
Across GPT-4, Claude, and other frontier models, none passed the test at fine-grained partitions. Direct estimates of quantities such as average US height disagreed with subgroup-conditioned estimates combined by population weights. The disagreement grows as partition granularity increases: more attributes, more leaves, larger gap.
Finding 2: The Macro Fallacy
Counterintuitively, finer subgroup-conditioned estimates are more accurate than direct population estimates. Ask the model for each of 10 neighborhoods' crime rates, then aggregate, and the result is closer to ground truth than asking directly for the citywide rate. The model holds rich subpopulation knowledge but cannot propagate it upward into a coherent aggregate estimate. Persona prompting amplifies the effect: a "25-year-old software engineer in San Francisco" yields a detailed, specific income figure whose aggregation outperforms the model's direct answer to "average income."
Finding 3: Implicit Prompting Is Insufficient
Carefully designed prompts that nudge the model toward probabilistic reasoning reduce but do not eliminate the inconsistency. This suggests the failure is architectural: knowledge is stored in contextual, persona-conditioned form rather than as an abstract probability distribution that the model can query coherently.
What This Reveals About LLM Knowledge Organization
Why It Matters
A New Reference-Free Evaluation Criterion
Traditional benchmarks require a ground-truth answer. Statistical self-consistency does not: if the model says X equals Y in one framing and X does not equal Y in another, the model is wrong regardless of the true value. This makes self-consistency a useful reference-free metric for AI alignment and safety research.
Takeaway
Frontier LLMs violate one of the simplest identities in probability theory when queries are reframed as partition-aggregate-then-marginalize. The Macro Fallacy indicates that persona-conditioned knowledge is more reliable than aggregated population knowledge, which is the opposite of what a coherent probabilistic model should exhibit. Closing this gap is an open architectural problem, not a prompting trick.