LLMs Know the Parts but Miscalculate the Whole: A Statistical Blind Spot Called the "Macro Fallacy"
> Paper: *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models* > arXiv: 2607.15277 > Authors: Patrik Wolf, Thomas Kleine Buening, Andreas Krause, Celestine Mendler-Dünner (ETH Zurich)
A Statistics Bureau That Can't Do Addition
Imagine a national statistics bureau with surveyors in every county. Each county surveyor reports: "62% of people in our county support policy X." The bureau's job is simple: compute a population-weighted average of the county numbers to get the national support rate.
But this bureau also has a department that runs a direct national survey. It reports 55%.
62% and 55% don't match. Someone made an error, or the biases of the two systems don't cancel out.
This is not hypothetical. This paper from ETH Zurich found that all current frontier LLMs are this "statistics bureau that can't do addition." Ask it "what fraction of Americans support X" and it gives one number; ask it to estimate views of "men," "women," "young people," and "older people" separately and aggregate with population weights, and you get a different number than the direct question.
How big is the gap? Surprisingly large. And — even more surprisingly — the aggregate-from-subgroups number is often closer to real poll data than the direct estimate.
This is what the paper calls the "macro fallacy": direct population-level estimates from LLMs are worse than estimates reconstructed from subpopulations.
Experimental Design: Three Steps
Step 1: Partition
Take a population (e.g., "US adults") and recursively split it with a binary tree. Level 0 is the whole. Level 1 splits by gender. Level 2 splits by age within each gender, and so on.
Each leaf node is a subpopulation described in natural language: "urban men aged 18–35," "rural women over 65," etc.
Step 2: Prompt
For every node in the tree (internal nodes and leaves), prompt the LLM in natural language: "What fraction of [subpopulation description] would answer 'yes' to [question]?" Record the model's output probability.
Step 3: Aggregate
Using the law of total probability, weight the leaf estimates by subpopulation size and aggregate upward to the root.
The law of total probability is the classic marble example: a bag has 60% red and 40% blue marbles; 30% of red and 70% of blue marbles have spots; overall spotted fraction = 0.6 × 0.3 + 0.4 × 0.7 = 0.46. This is one of the most basic identities in probability theory.
Then compare: the aggregated root estimate vs. the directly prompted root estimate. If the LLM were a consistent probability estimator, the two would be equal.
Findings: Systematic Violations of Consistency
Testing across multiple domains (GSS survey questions, demographic questions), multiple frontier models, and multiple tree structures, the paper finds:
All models broadly violate the law of total probability.
There is a systematic gap between direct and aggregated estimates. It is not random noise — it persists stably across tree structures and questions.
The "Macro Fallacy": The Most Counterintuitive Finding
This is the paper's core contribution. The usual assumption: asking directly about "Americans' views on X" should be more accurate than estimating each subpopulation and aggregating, since the latter accumulates errors across many estimates.
But the data show the opposite.
When the authors compare aggregated estimates to real polling data (GSS):
- Estimates reconstructed from finer-grained subpopulations are closer to ground truth than direct population-level estimates.
- The effect is stable across tree structures and estimation tasks.
- "Implicit prompting" — hinting at the aggregation structure in the prompt — can partially restore consistency.
Why Does This Happen?
The paper offers no definitive mechanism, but several hypotheses are worth discussing:
Hypothesis 1: Broad population prompts activate a "vague mode"
When you ask "What do Americans think of X?" the model may activate a "generic answer" mode — a fuzzy estimate that shrinks toward the center. Asking about "urban men aged 18–35" activates more precise estimates grounded in specific population patterns in training data. It's like asking a doctor "how long do people generally live" vs. "how long does a 65-year-old Asian male smoker generally live" — the specific patient profile retrieves more specific medical knowledge.
Hypothesis 2: LLMs are not Bayesian estimators
In-context learning is often interpreted as conditional inference: the prompt specifies context, and the model outputs an estimate of the corresponding conditional distribution. If that interpretation held, LLM estimates should satisfy the basic identities of probability. This paper shows that interpretation is structurally wrong. LLM outputs are not true conditional probabilities — they are text generation that *looks like* conditional probabilities but violates the structural constraints of probability theory.
This challenges the whole "LLM as approximate Bayesians" research paradigm. If your "approximate Bayesian" doesn't even satisfy the law of total probability, what exactly is it approximating?
Hypothesis 3: Training objectives don't reward consistency
LLMs are trained on next-token prediction, not on outputting distributions that satisfy probability axioms. Models were never explicitly trained to satisfy P(X) = Σ P(X|Y=y) P(Y=y), so failing this identity is expected from a training standpoint.
The surprising part is the direction of the finding: subpopulation estimates are more accurate than population estimates. This suggests the model learned specific knowledge about subgroups during training, but that knowledge gets systematically distorted during population-level "generalization."
What This Means
1. Don't use a single LLM call for population estimation
If you're using LLMs as a substitute for polls, market research, or demographic estimation — don't directly ask "what fraction of group X...". Instead: split the population into subpopulations, estimate each, and aggregate with size weights. Even though this involves more estimates, the final result is more accurate.
2. Self-consistency as a reference-free evaluation criterion
The paper introduces a new LLM evaluation dimension: statistical self-consistency. Most benchmarks require "correct answers" for comparison. Self-consistency doesn't — it only checks whether the model's internal estimates satisfy the axioms of probability. If A + B should equal C and the model says A + B ≠ C, you've found a bug even without knowing what C should be. It's a "free" evaluation dimension applicable to any meaningful partition, with no labeled data needed.
3. "LLMs as approximate Bayesians" needs re-examination
If LLM estimates violate the law of total probability, the theoretical framework interpreting in-context learning as "conditional inference" needs discounting. The model may be doing some kind of approximate conditional inference, but the approximation is much cruder than we assumed.
4. Practical implication: subpopulation prompt engineering
If you want more accurate demographic estimates from an LLM, give it specific subpopulation descriptions rather than broad group labels. It's not a sophisticated trick, but the principle behind it — the "macro fallacy" — is systematically revealed by this paper for the first time.
Limitations
1. Prior dependence: aggregation requires knowing subpopulation sizes (prior probabilities). The paper doesn't deeply explore sensitivity to prior misspecification. 2. Diagnosis, not treatment: the paper identifies the problem but proposes no fix. The "implicit prompting" in Section 5 only partially restores consistency and is treated briefly. 3. Arbitrariness of tree structure: which dimensions to split on, and how deep, remains somewhat arbitrary. The paper acknowledges but doesn't fully resolve this. 4. Mechanism unclear: why does the macro fallacy occur? The paper gives the phenomenon but not the mechanism.
Conclusion
This paper evokes an old philosophical question: is the whole equal to the sum of its parts? In probability theory, the answer is "yes" — the law of total probability guarantees it. In LLMs, the answer is "no" — the sum of the parts is systematically inconsistent with the whole.
More interesting still: the sum of the parts is often closer to the truth than the whole itself. It's as if LLMs retain fine-grained knowledge of the world at the micro level but lose information when "generalizing" to the macro level. The model's generalization ability becomes its weakness.
There's an echo of human cognition here: we are also good at judging specific cases but often err at statistical aggregation (Kahneman's System 1 / System 2 distinction). LLMs aren't imitating human reasoning — they're imitating human biases.
Next time you want an LLM to estimate a property of a population, ask yourself: should I ask about the whole, or ask about each part and add them up myself? The answer may be the latter.
---
Paper link: https://arxiv.org/abs/2607.15277 Authors' institution: ETH Zurich Code: no public code repository provided by the paper