Stochastic Sampling Is Epistemically Shallow: A Dimensionality Gap in LLM Diversity

Forum topic · 小凯 · 2026-07-25

Summary

A 2026 paper by Izhar Ali (Rowan University), accepted at the EIML Workshop at ICML 2026, challenges a common assumption in LLM evaluation: that multiple stochastic samples from a single model reveal meaningful epistemic structure. Using Marchenko-Pastur random matrix theory, the author compares a 100-sampling matrix from one model (Qwen2.5-7B, temperature=1) against a 24-model ensemble (temperature=0) across 500 questions. Results show internal sampling yields at most one eigenvalue above the noise floor, while cross-model evaluation exposes four. Ali attributes this to temperature acting as a uniform dilation of logits, producing unstructured noise rather than domain-specific knowledge signals. The paper introduces the phrase 'locally committed, globally incoherent' and discusses implications for selective prediction, hallucination detection, and ensemble learning, suggesting current sampling behavior resembles memorization more than understanding.

Paper

Title: Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs
Author: Izhar Ali (Rowan University)
arXiv: 2607.20464
Venue: EIML@ICML 2026 (2nd Workshop on Epistemic Intelligence in Machine Learning)

---

Background: Self-Consistency

Self-Consistency, introduced by Wang et al. (2023), samples multiple answers from a single LLM and takes a majority vote. Variants include answer-recurrence confidence, semantic entropy, and hallucination detection. All rely on the assumption that sampling variation encodes information about what the model knows versus guesses.

Core Question

Does within-model sampling expose the same kind of cross-question structure that a diverse model ensemble reveals?

Method: Marchenko-Pastur Random Matrix Theory

MP theory defines the eigenvalue spectrum expected from a purely random matrix. Eigenvalues exceeding the MP boundary indicate non-random structure. The author applies this as a mathematical microscope to distinguish real knowledge signals from noise.

Experimental Setup

Group A (within-model): 100 stochastic samples from Qwen2.5-7B on 500 questions (temperature=1) → 100×500 matrix
Group B (cross-model): 24 different LLMs (varied architectures, training data, scales), one deterministic run each (temperature=0) → 24×500 matrix

Key Findings

The Dimensionality Gap

Within-model matrix: at most one eigenvalue above the MP noise boundary, with a TW z-score ≤ +0.05 (statistically insignificant)
Cross-model matrix: four clear eigenvalues above the boundary, robustly validated against a matched Bernoulli null (500 Monte Carlo draws produced at most one signal eigenvalue)

The within-model diversity is effectively noise. The cross-model diversity carries structured epistemic content.

Temperature as Uniform Dilation

Temperature uniformly rescales logits across all tokens, regardless of domain, question type, or confidence. It cannot carve out the structural shadows needed to reveal ability dimensions. Sampling variations therefore lack the structured, cross-question correlations present in ensembles.

"Locally Committed, Globally Incoherent"

On edge cases, Qwen2.5-7B produced the same wrong answer with 87.4% probability across 100 samples, showing strong local commitment. However, these local commitments lacked cross-question correlation: errors on problem A and errors on problem B were atomized and unrelated, unlike human cognitive patterns where domain weaknesses produce correlated mistakes.

Implications for AI Engineering

1. Selective Prediction: Self-Consistency is fine for per-question uncertainty, but cannot identify systematic weakness categories that would guide targeted retraining. 2. Ensembles: Cross-model ensembles are not an expensive substitute for sampling depth; they are a fundamentally different information source exposing cognitive dimensions inaccessible within one model. 3. Hallucination Detection: Self-consistency-based detectors remain valid for local checks but miss deeper, systematic hallucination patterns.

Philosophical Note

The results raise the question of whether current LLMs exhibit memorization rather than understanding. True knowledge may require not just stability on individual questions but structured, predictable performance across related questions, where errors are correlated in diagnosable ways.

References

1. Ali, I. (2026). *Stochastic Sampling is Epistemically Shallow*. arXiv:2607.20464. 2. Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *ICLR*. 3. Lakshminarayanan, B., et al. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. *NeurIPS*. 4. Marčenko, V.A. & Pastur, L.A. (1967). Distribution of eigenvalues for some sets of random matrices. *Mathematics of the USSR-Sbornik*. 5. Farquhar, S., et al. (2024). Detecting Hallucinations in Large Language Models Using Semantic Entropy. *Nature*.