When Is Next-Token Prediction Useful? A Deep Dive into a Theoretical Indictment of LLM Training

Forum topic · 小凯 · 2026-05-25

Summary

This article offers a detailed commentary on Francesco Corielli's theoretical paper (arXiv:2605.23278), which asks under what conditions next-token prediction is actually useful. The paper distinguishes three objects: the full conditional language process governed by hidden non-textual environment variables, the marginalized text-only distribution, and the model-induced predictive distribution. Its central claim is that next-token prediction is useful only when the text prefix is a sufficient statistic for the hidden environment—formally, when the residual conditional mutual information I(X_{t+1}; Z_t | X_{≤t}) is approximately zero. This holds on 'local sufficiency islands' such as programming, math, and templated documents, but generally fails in open-ended factual domains, explaining why models excel at code yet hallucinate on facts. The framework reframes RAG and tool calling as sufficiency patches that only help when retrieved content actually captures the decisive environment variables, shows that temperature cannot recover missing information, and gives a structural account of synthetic-data contamination and model collapse. The commentary also covers mixture identifiability, prompt sensitivity, the limits of prompting, and the paper's self-acknowledged limitations as a purely theoretical work with no experiments.

This post is a deep-dive commentary on a theoretical paper challenging a foundational assumption of language modeling: that learning "given the preceding text, predict the next token" is a valid model of language understanding.

Paper information

| Field | Detail | |---|---| | Title | When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming | | Author | Francesco Corielli | | Affiliation | Independent researcher | | arXiv ID | 2605.23278 | | Date | May 22, 2026 | | Categories | cs.CL / stat.ML | | Core claim | Language models trained on corpora are not estimating the true conditional probability of the next token—real language generation is governed by non-textual environment variables. Next-token prediction is useful only when the text prefix is a sufficient statistic for the hidden environment. This condition approximately holds in domains like programming but generally fails in open factual domains. RAG and tool calling are patches for this condition, with strict sufficiency requirements of their own. |

Three objects, not one

The paper decomposes "what a language model does" into three distinct objects:

The full conditional language process $p_{\mathrm{full}}(x_{t+1} \mid x_{\leq t}, z_t)$ , where $$z_t$$ denotes non-textual environment variables: world facts, speaker intent, goals, beliefs, audience, task, social context.
The marginalized text-conditional distribution $p_{\mathrm{marg}}(x_{t+1} \mid x_{\leq t})$ , obtained by integrating out $$z_t$$ .
The model-induced predictive distribution $p_{\theta}(x_{t+1} \mid x_{\leq t})$ .

Treating one as another requires assumptions. Treating the model output as the marginalized distribution requires stationarity, ergodicity, and representativeness of the corpus—assumptions that are dubious for a mixed corpus spanning decades, domains, and genres.

Marginalization is not enough

$p_{\mathrm{marg}}(x_{t+1} \mid x_{\leq t}) = \int p_{\mathrm{full}}(x_{t+1} \mid x_{\leq t}, z)\, p(z \mid x_{\leq t})\, dz$

The marginalized distribution is useful only when $p_{\mathrm{full}} \approx p_{\mathrm{marg}}$ , i.e., when:

I(X_{t+1}; Z_t \mid X_{\leq t}) \approx 0

This residual conditional mutual information measures the remaining dependence between the next token and the hidden environment given the prefix. When it is large, the model merely plays a statistical game—generating text statistically compatible with the prefix but disconnected from reality. Example: asked to "give the correct diagnosis," a model outputs the diagnosis most frequent in training corpora after that phrase, while the actual patient's symptoms, test results, and history are entirely absent from the prefix. This reframes hallucination as structural information absence rather than insufficient model capability.

Local sufficiency islands

The training corpus is a mixture $\mathcal{D} = \sum_{k=1}^K \pi_k \mathcal{D}_k$ of domains with different hidden environments and different residual mutual informations $I_k(X_{t+1}; Z_t^{(k)} \mid X_{\leq t})$ .

Programming is the paradigmatic sufficiency island: explicit syntax, local constraints, specifiable requirements, externalized tests, textual error messages. A prefix like def merge_sort(arr): carries large amounts of latent state relevant to the next step.
Open factual domains ("the correct interpretation is", "the cause was") leave the prefix nearly uninformative about the hidden environment.

This explains why the same model can be surgical in code yet unreliable in factual reasoning: the two tasks have fundamentally different information structures.

Mixture identifiability

Learning the correct text-conditional distribution requires mixture components to be identifiable—prefixes must clearly indicate their domain. def merge_sort(arr) or SELECT customer_id FROM point to a component with probability near 1; "The answer is" leaves the posterior $p(k \mid x_{\leq t})$ diffuse, producing an averaged, cross-regime distribution that is statistically correct but practically useless.

This yields precise explanations for:

Prompt sensitivity: changing prompt wording shifts probability mass between mixture components, $p(k \mid x) \to p(k \mid x')$ .
Style–content confusion: the model recognizes genre but not epistemic situation.
Pseudo-authority: confident explanatory mode without the latent facts the mode presupposes.
Cross-regime contamination: cognitively inappropriate neighboring regimes bleed style and content into outputs when $p(k \mid x)$ is diffuse.

RAG and tool calls as sufficiency patches

With retrieved material $$R_t$$ and tool output $$A_t$$ , RAG is useful only when:

I_k(X_{t+1}; Z_t^{(k)} \mid X_{\leq t}, R_t, A_t) \approx 0

Retrieved text that does not capture the variables actually determining the next step is just extra paragraphs in the prompt—topically relevant, stylistically confident, epistemically inert. This explains why RAG works for course Q&A (where the material covers the epistemic environment) but fails for open factual queries (where the decisive causes are physical evidence, testimony, analysis—not retrievable text).

Temperature does not save you

p_{\theta,T}(i \mid x_{\leq t}) = \frac{\exp(\ell_i/T)}{\sum_j \exp(\ell_j/T)}

Temperature only changes sampling breadth; it does not restore missing environment variables. When the prefix is insufficient, higher temperature means higher-probability output of more extravagant hallucinations.

Structural contamination from synthetic data

With recursive training $P_{n+1} = (1-\alpha)P + \alpha Q_{\theta_n, T_n}$ , the problem is not merely that generated data is "low quality"— $Q_{\theta,T}$ is not a sample of the human language process at all. In regimes where sufficiency fails, model-generated text is a statistically plausible counterfeit. Model collapse (Shumailov et al.; Alemohammad et al.) acquires a new interpretation: each cycle injects insufficient-regime text as evidence of "correct language," amplifying precisely the parts where next-token prediction is least reliable.

The limits of prompting

Even if all relevant environment information is placed in the prompt, the model can only use conditional patterns it actually encountered in training. Prompts can select a learned conditional behavior—they cannot install one that was never learned. The same applies to RAG and tool outputs: they extend the conditioning sequence but act as effective conditioning variables only if the model acquired the corresponding pattern during training.

Honest limitations

Purely theoretical: no experiments, no benchmarks; the paper derives conditions that *should* hold, not measurements that they do.
Independent researcher: not stress-tested by large-lab peer pressure.
No scale analysis: larger models might indirectly bypass some sufficiency barriers; the framework does not predict this.
English-centric: multilingual mixture identifiability is not discussed.
No solutions: it is a precise problem statement; if hallucination is structural information absence, better language-model training alone cannot fix it.
Scope boundary the paper itself draws: it addresses whether models can learn correct text-conditional distributions, not whether high-probability continuations are true—the latter requires fact-checking, symbolic verification, external execution.

Stepping back: a quiet paradigm crack

The paper proposes no new model and runs no leaderboards. It asks, with information-theoretic precision, under what conditions the field's founding assumption is correct, and yields three groups of criteria:

Learnability: corpus stationarity, ergodicity, mixture identifiability.
Usefulness: the text prefix is a sufficient statistic for the hidden environment.
Verifiability: even when both hold, truth of generated continuations requires external checking.

Local sufficiency islands—programming, theorem proving, templated legal documents—satisfy these; open-world factual domains do not. The paper's contribution is not a fix but a set of criteria: when using next-token prediction as language understanding, check how large

I(X_{t+1}; Z_t \mid X_{\leq t})

actually is in your domain.

References

1. Corielli, F. (2026). When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming. *arXiv:2605.23278*. 2. Bender, E.M. & Koller, A. (2020). Climbing towards NLU. *ACL 2020*. 3. Shumailov, I., et al. (2024). The Curse of Recursion. *arXiv:2305.17493*. 4. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. *NeurIPS 2020*. 5. Shannon, C.E. (1948). A Mathematical Theory of Communication. *Bell System Technical Journal*.