Monologue in an Echo Chamber: Why Asking the Same AI 100 Times Never Yields the Truth

> Paper: Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs > Author: Izhar Ali (Rowan University) > arXiv: 2607.20464 > Venue: EIML@ICML 2026 (2nd Workshop on Epistemic Intelligence in Machine Learning)

---

🎲 Introduction: One Million Dice Rolls

Imagine facing an extremely complex puzzle. You don't know the answer, but there is a "sage" you can consult. You ask a question and get an answer. Unsure whether it's correct, you decide to ask the same sage the same question, each time asking him to reason along slightly different lines. You ask 100 times.

Now you have 100 answers. You tally them and find that answer #1 and answer #51 agree on 60% of the questions. You feel reassured: "If the same sage gives the same answer at different times, the answer is probably right."

But you've overlooked a key issue: those 100 answers, though slightly different, all come from the same sage, the same knowledge base, the same cognitive blind spot. If the sage has a systematic misunderstanding in some domain, then whether you ask 100 times or 1 million times, you'll get 1 million variants of the same misunderstanding.

This is the pointed question Izhar Ali raises at the ICML 2026 Epistemic Intelligence Workshop: when you sample an LLM 100 times (stochastic sampling), is the diversity you get diversity of "knowledge," or merely diversity of "noise"?

---

🧩 Chapter 1: Self-Consistency — A Seemingly Clever Shortcut

Before the paper itself, let's understand a widely used method: Self-Consistency.

Proposed by Wang et al. in 2023, the idea is remarkably simple: for a complex problem (e.g., math reasoning), instead of letting an LLM generate one answer, generate many and take a majority vote. If 7 of 10 answers say "42," confidence in "42" increases sharply.

The method works well in practice—cheap (no need for multiple models) and effective (significant accuracy gains on math benchmarks like GSM8K). It soon spawned variants:

• Answer reproduction confidence: how often the same answer recurs
• Semantic entropy: semantic equivalence, not just exact matches
• Hallucination detection: flagging likely hallucinations when sampled answers diverge widely

The shared assumption behind these methods: variation across samples from one model carries information about what the model "knows" and "doesn't know."

But Ali's question strikes at the core: does this variation carry structured knowledge information, or is it merely surface noise?

---

🏔️ Chapter 2: Two Kinds of Uncertainty

To understand the core finding, we need to distinguish two types of uncertainty estimation.

First: per-question uncertainty. This is self-consistency's home turf. For a specific question, generate multiple answers and measure disagreement. High disagreement = the model is uncertain. It's like flipping a coin repeatedly to estimate its bias—independent trials that jointly yield a precise estimate of "heads probability."

Second: cross-question structure. The deeper, more interesting question. Given a set of related math problems, if a model is weak at "algebra," we expect its performance to systematically degrade across all algebra problems. Such cross-question correlations reveal a model's "competence dimensions"—they say not just "I can't do this problem" but "I'm bad at this whole category."

A more everyday analogy:

• Per-question uncertainty is like asking a student to solve the same calculus problem 10 times to see if the answer is consistent.
• Cross-question structure is analyzing the student's performance in calculus, linear algebra, and probability, and discovering they systematically fail on "problems requiring spatial imagination"—a deeper cognitive signature.

Ali's core question: can self-consistency's repeated sampling reveal this cross-question structure?

---

🔬 Chapter 3: Random Matrix Theory — A Mathematical Microscope

To answer this, Ali borrows a powerful tool from physics and statistics: Marchenko-Pastur (MP) random matrix theory.

The core intuition is simple. Imagine a huge table where rows are "different attempts" and columns are "different questions," each cell being 0 (wrong) or 1 (right). With one attempt there's nothing to analyze, but 100 attempts give you a 100-row matrix.

Now the key question: of the correlations among those 100 rows, how much is genuine "structured knowledge signal" and how much is "random noise"?

MP theory gives an elegant answer: for a purely random matrix (every cell independently random), the eigenvalue spectrum follows a specific distribution—the MP distribution. Any eigenvalue beyond its boundary indicates "non-random structure."

It's a "noise baseline"—any signal above it deserves attention.

---

📊 Chapter 4: Experimental Design — 100 Samples vs. 24 Models

Ali designed an elegant controlled experiment.

Group A (within-model): Take one LLM (e.g., Qwen2.5-7B), run it 100 times per question (temperature = 1), record correctness each time. Result: a 100×500 matrix (100 runs, 500 questions).

Group B (cross-model): Take 24 different LLMs (different architectures, training data, scales), run each once (temperature = 0, deterministic), record correctness. Result: a 24×500 matrix.

Then apply the MP test to both matrices.

The results are striking:

In Group A—100 samples—the MP test found: at most one eigenvalue above the noise boundary. And even that signal was very weak (TW z-score ≤ +0.05, entirely within noise).

In other words: 100 samples of the same model contain almost no cross-question structured information.

But in Group B—24 different models—the MP test found four clear eigenvalues beyond the noise boundary. The result was rigorously validated: compared to a difficulty-matched independent Bernoulli null hypothesis (500 Monte Carlo draws), the null produces at most one signal eigenvalue, while Group B stably produces four.

This is the paper's "Dimensionality Gap":

• Within-model dimension: ≤ 1
• Cross-model dimension: ≥ 4

---

🌊 Chapter 5: Temperature Dilution — Why Sampling Is So "Shallow"

Why are 100 samples of one model so "impoverished"?

Ali offers a deep explanation: temperature is a uniform inflation of the logits.

Recall how an LLM generates text: it outputs a score (logit) for every possible next token, then a softmax converts scores into a probability distribution. The temperature T controls the distribution's sharpness:

• T = 0: fully greedy, always picks the highest-probability token
• T = 1: standard sampling
• T > 1: more "random," flatter distribution

Ali points out that temperature adjustment is a uniform operation on the whole distribution—it doesn't distinguish domains, question types, or "I know this" from "I'm guessing this." It's like a lamp that uniformly lights a room, casting no deeper shadows in some corners to reveal structure.

This yields a key conclusion: sampling variation induced by temperature is essentially "unstructured noise," not "structured knowledge difference."

A metaphor: imagine searching for books in a vast library. Temperature sampling is like taking a different random corridor in the same library each time—the corridor differs, but you always see the same library's books; you never suddenly see another library's collection. Using different models means visiting entirely different libraries.

---

🎯 Chapter 6: Locally Committed, Globally Incoherent

One particularly striking concept in the paper: "locally committed, globally incoherent."

Ali found that when Qwen2.5-7B errs on a borderline question, it picks the same wrong answer with 87.4% probability across 100 samples. The model has strong "local commitment"—it "firmly believes" a wrong answer is correct.

But crucially, these local commitments show no cross-question correlational structure. The model tends toward wrong answer X on question A and wrong answer Y on question B—with no systematic connection. Errors are "isolated events," not "symptom clusters."

This contrasts interestingly with human cognition. If a student repeatedly fails at "geometry requiring spatial imagination," we might diagnose a systematic weakness in spatial reasoning. Such cross-question correlation is the core of educational diagnosis. But LLM sampling errors seem to lack this structure—they are atomic, isolated, uncorrelated.

---

💡 Chapter 7: Implications for AI Engineering

1. Selective Prediction. If your system must decide "should I answer, or decline because I might be wrong," self-consistency is a good low-cost option—it gives accurate per-question uncertainty. But if you want to know "which categories of questions I systematically fail at and need extra training on," self-consistency can't help. You need genuine model diversity.

2. The value of model ensembles. Classic ensemble learning says "use multiple models to reduce variance." Ali's result gives a deeper reason: different models don't just reduce variance—they reveal "cognitive dimensions" inaccessible within a single model. A 24-model ensemble is not just "an expensive version of 100 samples"—it is a fundamentally different information source.

3. Limits of hallucination detection. Many current hallucination detectors rely on self-consistency—flagging likely hallucinations when sampled answers disagree. Ali's result doesn't deny this method's validity (per-question uncertainty is indeed accurate), but it reminds us: such methods cannot reveal deeper, cross-question systematic hallucination patterns.

---

🌌 Chapter 8: A Philosophical Coda — What Is "Knowing"?

Though technical, the paper touches a profound question: what does it mean to "know"?

Self-consistency assumes: if a model gives consistent answers across samples, it "knows" the answer. But Ali's results suggest consistency may be mere "local inertia"—the model is trained well enough on a specific question that sampling noise can't dislodge it. That isn't "understanding."

Real "knowing" may require a stronger condition: not only stability on specific questions, but predictable structure across related questions. A model that "knows" algebra should show correlated patterns across all algebra problems—its errors should be systematic, structured, diagnosable, not random.

In this sense, Ali's dimensionality gap reveals a disquieting possibility: current LLM sampling behavior may be closer to "memory" than "understanding"—models memorize vast numbers of isolated facts and patterns, but those memories lack deep structured interconnection.

---

🔮 Conclusion: From Echo Chamber to Symphony Orchestra

Return to the opening metaphor. You asked the same sage the same question 100 times and got 100 slightly different answers. You thought you had 100 independent perspectives; in fact, you had 100 echoes of one perspective.

The paper's value lies in quantifying this intuition with rigorous mathematical tools. Self-consistency is not a cheap substitute for an ensemble—it is a fundamentally different, informationally impoverished approximation.

For AI researchers and engineers, the finding is both warning and direction:

• Warning: don't over-rely on sampling's "surface diversity"
• Direction: in pursuing genuine epistemic dimensionality, model diversity may be more valuable than sampling depth

Just as a recording of one instrument played 100 times can never replace the harmony and dialogue among the different instruments of a symphony orchestra, true wisdom may lie not in how loud one voice is, but in how multiple voices resonate.

---

📚 References

1. Ali, I. (2026). *Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs*. arXiv:2607.20464. EIML@ICML 2026. 2. Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *ICLR*. 3. Lakshminarayanan, B., et al. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. *NeurIPS*. 4. Marčenko, V.A., & Pastur, L.A. (1967). Distribution of eigenvalues for some sets of random matrices. *Mathematics of the USSR-Sbornik*. 5. Farquhar, S., et al. (2024). Detecting Hallucinations in Large Language Models Using Semantic Entropy. *Nature*.

---

*Analysis completed July 26, 2026*