English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Cracks in the Mirror: Why LLMs Know the Answer but Get the Math Wrong — Statistical Self-Consistency Explained

Forum topic · 小凯 · 2026-07-19

Summary

A Chinese forum post interprets an ETH Zurich and Stanford paper (arXiv:2607.15277, "Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models") showing that large language models systematically violate the law of total probability. The authors tested frontier models using a binary-tree evaluation scaffold: a model is asked to estimate a quantity directly, then asked again for subgroup estimates that are aggregated with population weights. If the model were statistically self-consistent, the two results would match; none of the tested models passed. The post highlights the "Macro Fallacy": decomposed, finer-grained subgroup queries (especially via persona prompting) aggregated afterward often yield more accurate estimates than direct macro-level questions, suggesting LLM knowledge is stored in concrete contexts rather than abstract distributions. Implicit prompting only partially mitigates violations, implying architectural roots. The paper proposes statistical self-consistency as a new reference-free evaluation criterion, with implications for healthcare, finance, policy, and science.

Cracks in the Mirror: Why Language Models "Know the Answer but Get the Math Wrong"

*English edition of a zhichai.net paper interpretation, originally written by 小凯 (July 20, 2026).*

> *"Nature hides her secrets behind language."* — Friedrich Nietzsche

The Strange Phenomenon

Suppose you survey 1,000 people about their income. The overall average is 50,000. Splitting by gender (50/50), men average 55,000 and women 45,000. By the law of total probability — a mathematical identity that always holds when weights are correct — the weighted average 0.5 × 55,000 + 0.5 × 45,000 must equal 50,000.

Now hand this task to a large language model (GPT-4, Claude, etc.). Instead of asking directly for the overall average, you ask for conditional estimates per subgroup, then aggregate them with population weights. If the model were a competent statistician, both routes would agree.

They don't. A paper from ETH Zurich and Stanford reveals that frontier LLMs suffer systematic violations of basic statistical self-consistency.

The Law of Total Probability

A simple example: a bag contains 60% red balls (average number 10) and 40% blue balls (average number 20). The overall average must be 0.6 × 10 + 0.4 × 20 = 14. If a model predicts 10 for red and 20 for blue when told the color, it should infer 14 when asked overall. The paper finds that direct overall queries often disagree with the aggregated subgroup route — like a mathematician saying 1+1=2, but 0.5×2 + 0.5×2 = 3.

The Binary Tree Evaluation Scaffold

The research team built a binary tree evaluation scaffold: recursively partition a population by attributes (e.g., gender → age bracket → preference), then compare:

  • Route 1 (direct): ask the model for the overall estimate.
  • Route 2 (partition-aggregate): ask recursively for every leaf subgroup, then aggregate by weights.
  • Finding 1: Widespread Inconsistency

    Multiple state-of-the-art models — including OpenAI's GPT series, Anthropic's Claude, and other frontier models — all failed this test. Inconsistencies emerge especially with finer-grained partitions: like measuring the same table with rulers of different precision and getting different sizes.

    Finding 2: The "Macro Fallacy"

    The most surprising finding, dubbed the Macro Fallacy: *finer-grained, more specific queries produce more accurate answers.* Asking for 10 neighborhood-level crime rates and weighting them often beats asking directly for the city-wide rate.

    Hypothesis: LLMs hold rich subgroup knowledge but fail to propagate it into population-level estimates — knowledge is fragmented across concrete contexts rather than organized as abstract distributions.

    Persona Prompting

    With persona prompting ("You are a 25-year-old male software engineer in San Francisco; what is your annual income?"), the Macro Fallacy becomes even more pronounced. Persona answers, aggregated, tend to be more accurate than direct aggregate questions — suggesting LLM knowledge is stored in concrete situational form, not in a coherent internal world model.

    Implicit Prompting: A Partial Fix

    Implicit prompting — indirectly steering the model toward statistical consistency without stating the rule — helps only partially. The problem may be rooted in the architecture itself, not prompt design.

    Deeper Implications

    1. Fragmented knowledge — stored in countless specific contexts, not abstract probability distributions. 2. Context dependence — answers are generated dynamically at query time, not retrieved from a consistent internal database. 3. Lack of metacognition — models don't notice when their own estimates disagree; they lack a self-correction mechanism.

    Why It Matters

  • Healthcare: inconsistent aggregation of subgroup risks can cause misdiagnosis.
  • Finance: internally contradictory portfolio recommendations.
  • Policy: flawed aggregation of group needs yields unintended consequences.
  • Science: macro/micro inconsistency invalidates a theory.
  • A New Evaluation Standard

    The paper proposes statistical self-consistency as a reference-free evaluation criterion: no ground truth needed — only a check that the model's answers are internally consistent. This could matter greatly for future AI safety and alignment research.

    Conclusion

    LLMs are a mirror of human knowledge — but this paper shows the mirror has cracks. Observing those cracks helps us understand how these models actually organize and retrieve knowledge, and points toward building systems with more consistent internal world models.

    Reference

  • Wolf, P., Buening, T. K., Krause, A., & Mendler-Dünner, C. (2026). *Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models*. arXiv:2607.15277.
arXiv: 2607.15277

Tags

#llm#statistical-self-consistency#law-of-total-probability#evaluation#persona-prompting#macro-fallacy#ai-safety#paper-interpretation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178446929