LLMs Know the Parts but Misjudge the Whole: A Statistical Blind Spot Called the "Macro Fallacy"
> Paper: Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models > arXiv: 2607.15277 > Authors: Patrik Wolf, Thomas Kleine Buening, Andreas Krause, Celestine Mendler-Dünner (ETH Zurich)
A Statistics Bureau That Can't Do Addition
Imagine a statistics bureau with surveyors in every county. Each surveyor reports: "62% of people in our county support policy X." The bureau's job is simple: compute the population-weighted average to get the national rate.
But this bureau also has a second department that runs a direct national survey. Its number: 55%.
62% vs. 55%. Someone is wrong, and the biases of the two systems don't cancel out.
This is not hypothetical. A paper from ETH Zurich found that every current frontier LLM behaves like this "statistics bureau that can't do addition." Ask it "what share of Americans support X" and you get one number; ask it to estimate men, women, young people, older people separately and aggregate with population weights, and you get a different number.
And the difference is surprisingly large. More surprisingly still, the bottom-up aggregated estimate is often closer to real polling data than the direct one.
This is what the paper calls the "macro fallacy": LLMs' direct group-level estimates are worse than estimates reconstructed from subgroups.
The Experimental Design: Three Steps
Step 1: Partition
Take a population (e.g., "US adults") and recursively split it with a binary tree. Level 0 is the whole. Level 1 splits by gender; level 2 splits each gender by age, and so on. Each leaf is a subpopulation described in natural language: "urban men aged 18–35," "rural women over 65," etc.
Step 2: Prompt
For every node in the tree (internal and leaf), prompt the LLM: "What proportion of [subgroup description] would answer 'yes' to [question]?" Record the model's output probability.
Step 3: Aggregate
Using the law of total probability, aggregate leaf estimates weighted by subgroup size, layer by layer, up to the root.
The law of total probability is the marble example: a bag is 60% red and 40% blue; 30% of red and 70% of blue marbles are speckled; total speckled = 0.6 × 0.3 + 0.4 × 0.7 = 0.46. It's one of the most basic identities in probability theory.
Then compare: the aggregated root estimate vs. the directly prompted root estimate. If the LLM were a consistent probability estimator, they'd be equal.
Findings: Systematic Violations of Consistency
Across multiple domains (GSS survey questions, demographic questions), multiple frontier models, and multiple tree structures, the paper finds:
All models broadly violate the law of total probability.
There is a systematic gap between direct and aggregated estimates. It is not random noise—it persists stably across tree structures and questions.
The "Macro Fallacy": The Most Counterintuitive Finding
This is the paper's core contribution. The usual assumption: directly asking "what do Americans think of X" should be more accurate than asking each subgroup and aggregating, since the latter accumulates estimation errors.
But the data show the opposite.
When the authors compare aggregated estimates against real polling data (GSS):
- Estimates reconstructed from finer-grained subgroups are closer to ground truth than direct group-level estimates.
- The effect is stable across tree structures and estimation tasks.
- "Implicit prompting"—hinting at the aggregation structure in the prompt—partially restores consistency.
Why Does This Happen?
The paper offers no definitive mechanism, but several hypotheses are worth discussing:
Hypothesis 1: Group-level prompts activate a "vague mode"
When asked "what do Americans think of X," the model may activate a generalizing mode—producing a fuzzy, shrunk-toward-the-middle estimate. A specific description like "urban men aged 18–35" activates more precise, pattern-grounded estimates.
It's like asking a doctor "how long do people generally live" vs. "how long does a 65-year-old Asian male smoker generally live"—the specific patient profile activates more specific medical knowledge.
Hypothesis 2: LLMs are not Bayesian estimators
In-context learning is often interpreted as conditional inference: the prompt specifies context, and the model outputs the corresponding conditional distribution estimate. If that interpretation held, LLM estimates should satisfy the basic identities of probability.
This paper shows that interpretation is structurally wrong. LLM outputs are not true conditional probabilities—they are text generation that *looks like* conditional probabilities but doesn't obey probability's structural constraints. This challenges the whole "LLM as approximate Bayesians" research paradigm: if your approximate Bayesian doesn't even satisfy the law of total probability, what exactly is it approximating?
Hypothesis 3: Training objectives don't reward consistency
LLMs are trained on next-token prediction, not on outputting distributions that satisfy probability axioms. They were never explicitly trained to satisfy P(X) = Σ P(X|Y=y) P(Y=y). From a training perspective, the violation is expected.
The surprising part is the direction of the finding: subgroup estimates are *more accurate*. The model has learned concrete knowledge about subgroups, but that knowledge gets systematically distorted in the process of group-level "generalization."
What This Means
1. Don't use single LLM calls for population estimates
If you use LLMs for polling substitutes, market research, or demographic estimation—don't directly ask "what proportion of group X..." Instead, split the population into subgroups, estimate each, and aggregate with size weights. Even with more estimation calls, the result is more accurate.
2. Self-consistency as a reference-free evaluation criterion
The paper introduces a new evaluation dimension: statistical self-consistency. Most benchmarks need "correct answers" to compare against. Self-consistency doesn't—it only checks whether the model's internal estimates satisfy probability axioms. If A + B should equal C and the model says A + B ≠ C, you've found a bug even without knowing what C should be. It's a "free" evaluation dimension—applicable to any meaningful partition, no labeled data required.
3. "LLM as approximate Bayesians" needs re-examination
If LLM estimates don't satisfy the law of total probability, the theoretical framework interpreting in-context learning as conditional inference must be discounted. The model may be doing something like approximate conditional inference—but the approximation is far cruder than assumed.
4. Practical takeaway: subgroup prompt engineering
If you want more accurate demographic estimates from an LLM, give it specific subgroup descriptions rather than broad group labels. Not a deep technique—but the principle behind it, the "macro fallacy," is what this paper systematically reveals for the first time.
Limitations
1. Prior dependence: Aggregation requires knowing subgroup sizes (prior probabilities). Sensitivity to misspecified priors isn't explored in depth. 2. Diagnosis, not cure: The paper identifies the problem but offers no fix. "Implicit prompting" in Section 5 only partially restores consistency and is treated briefly. 3. Arbitrariness of tree structure: Which dimensions to split on, and how deep, remain somewhat arbitrary. The paper acknowledges but doesn't fully resolve this. 4. Unclear mechanism: Why does the macro fallacy occur? The paper gives the phenomenon, not the mechanism.
Closing Thoughts
This paper evokes an old philosophical question: is the whole equal to the sum of its parts? In probability theory, yes—the law of total probability guarantees it. In LLMs, no—the sum of the parts is systematically inconsistent with the whole.
More interestingly, the sum of the parts is often closer to the truth than the whole itself. It's as if LLMs retain fine-grained world knowledge at the micro level but lose information when "generalizing" at the macro level. The model's ability to generalize turns out to be its weakness.
There's an echo of human cognition here: we're also good at judging specific cases but frequently err at statistical aggregation (Kahneman's System 1 vs. System 2). LLMs aren't imitating human reasoning—they're imitating human biases.
Next time you want an LLM to estimate a population attribute, ask yourself: should I ask about the whole, or about each part and add it up myself? The answer is probably the latter.
---
Paper link: https://arxiv.org/abs/2607.15277 Author affiliation: ETH Zurich Code: No public code repository provided by the paper