Cracks in the Mirror: Why LLMs 'Know the Answer but Can't Do the Math'

> *"Nature hides her secrets behind language."* — Friedrich Nietzsche

This post is an English edition of a Chinese forum walkthrough of the ETH Zurich / Stanford paper *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models* (Wolf, Buening, Krause, Mendler-Dünner, arXiv:2607.15277).

A Strange Phenomenon

Imagine a survey: the national average income is 50,000, and the averages for men and women are 55,000 and 45,000. Weighting by population shares (0.5 × 55,000 + 0.5 × 45,000) recovers 50,000 exactly. This is the Law of Total Probability—an ironclad mathematical identity.

Now give the same task to an LLM: ask it for subgroup estimates, then weight and aggregate them. If the model were a competent statistician, the result should match its direct estimate of the population average. The paper shows it does not—systematically.

The Law of Total Probability

The law is basic as 1+1=2: if a bag is 60% red balls (average value 10) and 40% blue balls (average value 20), the overall average must be 0.6 × 10 + 0.4 × 20 = 14. The paper finds that when models are asked directly for the overall average, their answers often disagree with the value computed from their own subgroup estimates.

The Binary Tree Evaluation Scaffold

The researchers built a decision tree: partition a population by gender, then age, then preferences, and so on. They compare:

• Direct estimation: "What is the average preference across all 1,000 guests?"
• Partition-aggregate: recursively ask about each leaf subgroup, then weight the answers by group sizes.

A statistically self-consistent model must produce identical results under both methods.

Key Findings

• Widespread violations: None of the tested frontier models (GPT series, Claude, and other leading LLMs) pass the test. With fine-grained partitions, aggregated subgroup estimates diverge from direct macro estimates.
• The Macro Fallacy: counterintuitively, decomposing a question into finer subgroups yields answers that aggregate to something *more accurate* than the model's direct macro-level estimate.
• Persona prompting: asking the model to role-play a specific person (e.g., "a 25-year-old male software engineer in San Francisco") activates richer subpopulation knowledge, making the macro fallacy even more pronounced. LLM knowledge appears stored in concrete contexts rather than abstract distributions.
• Implicit prompting: indirectly nudging the model toward statistical consistency helps only partially—the flaw likely runs deeper than prompt design.

Deeper Implications

1. Fragmented knowledge: models hold many "stories" about subgroups but cannot integrate them into a coherent whole. 2. Context dependence: answers are dynamically generated per query rather than retrieved from a consistent internal world model. 3. Lack of metacognition: models do not notice that their own estimates are inconsistent, so they cannot self-correct.

Why It Matters

Statistical self-consistency failures could be critical in healthcare (risk aggregation for diagnosis), finance (internally contradictory portfolio advice), policy analysis (aggregating stakeholder needs), and scientific inference (consistency is a basic requirement).

The paper proposes statistical self-consistency as a reference-free evaluation criterion: unlike traditional benchmarks, it needs no ground truth—if a model's macro estimate contradicts its own aggregated sub-estimates, the model is provably flawed regardless of the true value. This could become an important tool for AI safety and alignment research.

> "The first principle is that you must not fool yourself—and you are the easiest person to fool." — Richard Feynman

References

• Wolf, P., Buening, T. K., Krause, A., & Mendler-Dünner, C. (2026). *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models*. arXiv:2607.15277.