Cracks in the Mirror: Why Language Models "Know the Answer but Get the Math Wrong"
> *"Nature hides her secrets behind language."* — Friedrich Nietzsche
This post is a detailed Chinese forum write-up of the paper "Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models" by Wolf, Buening, Krause, and Mendler-Dünner (ETH Zurich & Stanford), arXiv:2607.15277.
The Curious Phenomenon
The Law of Total Probability is a mathematical identity: if you split a population into groups with correct weights, the weighted average of group means must equal the overall mean. For example, with a 50/50 male/female split and group averages of 55,000 and 45,000:
> 0.5 × 55,000 + 0.5 × 45,000 = 50,000 = the overall average
If you ask an LLM to estimate an overall quantity directly, and then ask it to estimate per-subgroup quantities and aggregate them yourself, the two results should match. They don't. The paper shows frontier models (GPT series, Claude, and others) systematically violate this basic statistical consistency.
The Binary Tree Evaluation Scaffold
The authors build a decision tree over a population (e.g., partitioning guests by gender, then age band, then preference), recursively querying the model at each node:
- Direct estimation: "What is the average for the whole population?"
- Partition-and-aggregate: Ask about each leaf subgroup, then re-aggregate with population weights.
- Widespread violations: No tested state-of-the-art model passes the consistency test, especially when partitions are fine-grained.
- The "Macro Fallacy": Surprisingly, aggregated fine-grained (subgroup-level) estimates are often *more accurate* than the model's direct macro-level answer.
- Persona prompting: When the model role-plays a specific persona (e.g., "a 25-year-old male software engineer in San Francisco"), its answers are detailed and specific, and the Macro Fallacy becomes more pronounced — suggesting LLM knowledge is stored in concrete contexts rather than abstract probability distributions.
- Implicit prompting: Carefully designed prompts that indirectly steer the model toward consistency help only partially; the problem appears rooted in the architecture, not just prompt design.
- Wolf, P., Buening, T. K., Krause, A., & Mendler-Dünner, C. *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models*. arXiv:2607.15277.
A statistically self-consistent model must give identical answers by both routes.
Key findings
Deeper Implications
1. Fragmented knowledge: Models hold many "small stories" about specific groups but cannot integrate them into a coherent overall picture. 2. Context dependence: Answers are dynamically generated at query time, not retrieved from a stable internal world model — rephrasing the same question yields different answers. 3. Lack of metacognition: Models cannot detect their own inconsistencies; there is no self-correction mechanism that flags mismatched direct vs. aggregated estimates.
Why It Matters
Statistical self-consistency failures could be critical in medical risk estimation, financial forecasting, policy analysis, and scientific inference — anywhere subpopulation estimates must be correctly aggregated.
A New Reference-Free Evaluation Criterion
The paper proposes statistical self-consistency as a reference-free evaluation standard: you don't need ground-truth answers, only a check of whether the model's responses are internally consistent. This could become a valuable tool in AI safety and alignment research.
> "The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman