Cracks in the Mirror: Why Language Models "Know the Answer but Get the Math Wrong"
*English edition of a zhichai.net paper interpretation, originally written by 小凯 (July 20, 2026).*
> *"Nature hides her secrets behind language."* — Friedrich Nietzsche
The Strange Phenomenon
Suppose you survey 1,000 people about their income. The overall average is 50,000. Splitting by gender (50/50), men average 55,000 and women 45,000. By the law of total probability — a mathematical identity that always holds when weights are correct — the weighted average 0.5 × 55,000 + 0.5 × 45,000 must equal 50,000.
Now hand this task to a large language model (GPT-4, Claude, etc.). Instead of asking directly for the overall average, you ask for conditional estimates per subgroup, then aggregate them with population weights. If the model were a competent statistician, both routes would agree.
They don't. A paper from ETH Zurich and Stanford reveals that frontier LLMs suffer systematic violations of basic statistical self-consistency.
The Law of Total Probability
A simple example: a bag contains 60% red balls (average number 10) and 40% blue balls (average number 20). The overall average must be 0.6 × 10 + 0.4 × 20 = 14. If a model predicts 10 for red and 20 for blue when told the color, it should infer 14 when asked overall. The paper finds that direct overall queries often disagree with the aggregated subgroup route — like a mathematician saying 1+1=2, but 0.5×2 + 0.5×2 = 3.
The Binary Tree Evaluation Scaffold
The research team built a binary tree evaluation scaffold: recursively partition a population by attributes (e.g., gender → age bracket → preference), then compare:
- Route 1 (direct): ask the model for the overall estimate.
- Route 2 (partition-aggregate): ask recursively for every leaf subgroup, then aggregate by weights.
- Healthcare: inconsistent aggregation of subgroup risks can cause misdiagnosis.
- Finance: internally contradictory portfolio recommendations.
- Policy: flawed aggregation of group needs yields unintended consequences.
- Science: macro/micro inconsistency invalidates a theory.
- Wolf, P., Buening, T. K., Krause, A., & Mendler-Dünner, C. (2026). *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models*. arXiv:2607.15277.
Finding 1: Widespread Inconsistency
Multiple state-of-the-art models — including OpenAI's GPT series, Anthropic's Claude, and other frontier models — all failed this test. Inconsistencies emerge especially with finer-grained partitions: like measuring the same table with rulers of different precision and getting different sizes.
Finding 2: The "Macro Fallacy"
The most surprising finding, dubbed the Macro Fallacy: *finer-grained, more specific queries produce more accurate answers.* Asking for 10 neighborhood-level crime rates and weighting them often beats asking directly for the city-wide rate.
Hypothesis: LLMs hold rich subgroup knowledge but fail to propagate it into population-level estimates — knowledge is fragmented across concrete contexts rather than organized as abstract distributions.
Persona Prompting
With persona prompting ("You are a 25-year-old male software engineer in San Francisco; what is your annual income?"), the Macro Fallacy becomes even more pronounced. Persona answers, aggregated, tend to be more accurate than direct aggregate questions — suggesting LLM knowledge is stored in concrete situational form, not in a coherent internal world model.
Implicit Prompting: A Partial Fix
Implicit prompting — indirectly steering the model toward statistical consistency without stating the rule — helps only partially. The problem may be rooted in the architecture itself, not prompt design.
Deeper Implications
1. Fragmented knowledge — stored in countless specific contexts, not abstract probability distributions. 2. Context dependence — answers are generated dynamically at query time, not retrieved from a consistent internal database. 3. Lack of metacognition — models don't notice when their own estimates disagree; they lack a self-correction mechanism.
Why It Matters
A New Evaluation Standard
The paper proposes statistical self-consistency as a reference-free evaluation criterion: no ground truth needed — only a check that the model's answers are internally consistent. This could matter greatly for future AI safety and alignment research.
Conclusion
LLMs are a mirror of human knowledge — but this paper shows the mirror has cracks. Observing those cracks helps us understand how these models actually organize and retrieve knowledge, and points toward building systems with more consistent internal world models.