When Is Next-Token Prediction Useful? A Theory Paper's Indictment of LLM Training

Forum topic · 小凯 · 2026-05-25

Summary

An independent researcher, Francesco Corielli, has posted a purely theoretical paper (arXiv:2605.23278) arguing that language models trained via next-token prediction are not estimating true conditional probabilities of language, because real language generation is governed by hidden non-textual environment variables (facts, intent, goals, context). The paper formalizes the gap between the full conditional process, the marginalized text-only distribution, and the model's predictive distribution, introducing a 'residual conditional mutual information' criterion: next-token prediction is useful only when the text prefix is approximately a sufficient statistic for the hidden environment. This holds on 'local sufficiency islands' like programming and formal domains, but generally fails in open-world factual tasks, explaining why the same model can be precise at coding yet hallucinate on factual questions. The paper reframes RAG and tool calling as patches that must themselves satisfy strict sufficiency conditions, argues temperature cannot restore missing information, offers a new interpretation of synthetic-data model collapse, and shows prompts cannot substitute for missing training experience. A Chinese tech forum post provides a detailed structured解读, noting the paper's limitations: it is purely theoretical, unvalidated by experiments, and offers no solutions.

When Is Next-Token Prediction Useful? — A Deep Dive

| Paper info | | |---|---| | Title | When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming | | Author | Francesco Corielli | | Affiliation | Independent researcher | | arXiv ID | 2605.23278 | | Date | May 22, 2026 | | Categories | cs.CL / stat.ML | | Core claim | Language models trained on corpora are not estimating the true conditional probability of the next token — real language generation is governed by non-textual environment variables. Next-token prediction is useful only when the text prefix is a sufficient statistic for the hidden environment. This holds approximately in domains like programming but generally fails in open factual domains. RAG and tool calling are patches for this condition and have strict sufficiency requirements themselves. |

Key points

The whole world runs on an assumption so simple that nobody questions it: language models learn the probability of the next word given the preceding text. Shannon defined it in 1948, Bengio did it in 2003, and GPT-4 did it in 2024. Almost nobody has asked: under what conditions is it correct?

Francesco Corielli, a researcher with no institutional affiliation, wrote a paper with no experiments, no tables, and no SOTA numbers. It just poses the question and pushes it to precise information-theoretic limits.

Three different objects

The paper splits "what a language model does" into three objects:

The full conditional language process $p_{\mathrm{full}}(x_{t+1} \mid x_{\leq t}, z_t)$ , where $$z_t$$ denotes hidden non-textual environment variables: world facts, speaker intent, goals, beliefs, audience, task, social context.
The marginal text-conditional distribution $p_{\mathrm{marg}}(x_{t+1} \mid x_{\leq t})$ — what remains after integrating out $$z_t$$ .
The model-induced predictive distribution $p_{\theta}(x_{t+1} \mid x_{\leq t})$ — what the model actually outputs.

Treating one object as another requires assumptions at every step. Treating model output as the marginal requires corpus stationarity, ergodicity, and representativeness — assumptions that are dubious to the point of being近乎 comical when applied to a mixed corpus scraped from the internet, spanning fifty years, hundreds of domains and genres. And the model doesn't fit a conditional probability table — it fits parametric compression regularities, with smooth generalization across approximate contexts.

Marginalization alone is not enough

p_{\mathrm{marg}}(x_{t+1} \mid x_{\leq t}) = \int p_{\mathrm{full}}(x_{t+1} \mid x_{\leq t}, z) p(z \mid x_{\leq t}) dz

The marginal is useful only when $p_{\mathrm{full}} \approx p_{\mathrm{marg}}$ , i.e. when:

I(X_{t+1}; Z_t \mid X_{\leq t}) \approx 0

This residual conditional mutual information measures how much hidden-environment information remains relevant after conditioning on the prefix. When it is large, the model is just playing a statistical game: generating text statistically compatible with the prefix, unrelated to reality.

The chilling example: you ask a model "the correct diagnosis is" and it writes a disease name — the statistically most likely continuation in the training corpus. But the actual patient in front of you — symptoms, lab results, history — none of it entered the prefix. The residual mutual information is enormous, yet the model knows nothing about it. This pushes the root of hallucination from "the model isn't big enough" to structural information absence — the training paradigm cannot reach that information from the root.

Local sufficiency islands

The training corpus is a mixture:

\mathcal{D} = \sum_{k=1}^K \pi_k \mathcal{D}_k

Each component has different hidden environments and different sufficiency conditions $I_k(X_{t+1}; Z_t^{(k)} \mid X_{\leq t})$ . This yields local sufficiency islands — domains where the text prefix nearly covers the hidden environment.

Programming is the paradigm case: code syntax is explicit, constraints are local, specifications can be written in comments, tests externalize hidden behavior, error messages literalize runtime state. def merge_sort(arr): identifies the programming regime *and* carries much of the latent state relevant to the next step.

But prefixes like "the best response to this policy is" or "the cause of the accident was" tell you almost nothing. This explains the perplexing behavior pattern of language models: surgical precision in programming, nonsense in factual reasoning — because the two tasks have fundamentally different information structures.

Mixture identifiability

Learnability and usefulness are separate questions. Learning the correct text-conditional distribution requires the mixture components to be identifiable — the prefix must clearly indicate which regime it belongs to. def merge_sort(arr) or SELECT customer_id FROM point to programming with near-certainty; "The answer is" leaves the regime posterior $p(k \mid x_{\leq t})$ diffuse, so the model outputs a blurred, averaged, cross-regime mixture — statistically correct, practically useless.

This yields precise explanations for:

Prompt sensitivity: tweaking prompt wording shifts probability mass between mixture components ( $p(k \mid x) \to p(k \mid x')$ ), not "the model's understanding."
Style–content confusion: the model recognizes the genre but not the epistemic situation — the right tone with possibly wrong content.
Pseudo-authority: high-confidence explanatory mode without the latent factual state that mode presupposes.
Cross-regime contamination: statistically similar but epistemically inappropriate regimes bleed style and content into outputs when $p(k \mid x)$ is diffuse.

RAG and tools: sufficiency patches

RAG and tool calling are reframed — not "giving the model more knowledge," but converting missing environment variables into text. With retrieval $$R_t$$ and tool output $$A_t$$ , RAG is useful only when:

I_k(X_{t+1}; Z_t^{(k)} \mid X_{\leq t}, R_t, A_t) \approx 0

If the retrieved text doesn't contain the variables that actually determine the next step, it's just extra paragraphs in the prompt. This explains why RAG works well for course QA (where the materials genuinely cover the epistemic environment) but fails on open factual queries like "what caused the accident" — determined by physical evidence, testimony, and mechanics, not retrievable text.

Temperature won't save you

p_{\theta,T}(i \mid x_{\leq t}) = \frac{\exp(\ell_i/T)}{\sum_j \exp(\ell_j/T)}

Temperature only changes sampling breadth. It doesn't restore missing environment variables. When the prefix is sufficient, moderate temperature helps explore valid continuations; when it isn't, high temperature means outputting wilder hallucinations with higher probability.

Structural contamination from synthetic data

With recursion into training, $P_{n+1} = (1-\alpha)P + \alpha Q_{\theta_n, T_n}$ . The problem isn't that generated data is "low quality" — it's that $Q_{\theta,T}$ is not a sample of the human language process at all. In regimes where the residual mutual information isn't near zero, every generated passage is a statistically plausible counterfeit. Model collapse (Shumailov et al., Alemohammad et al.) gets a new interpretation: collapse isn't just loss of distribution tails — it's injecting sufficiency-deficient text as evidence of "correct language," amplifying the regime where next-token prediction is least reliable.

The limit of prompts: you can't teach what the model never saw

Even if you write all relevant environment information into a prompt, the model can see it — but can it use it? Not guaranteed. The model can only exploit "conditional information → output continuation" patterns it actually encountered in training. If the corpus never contained many instances of "programming to a full spec given the spec," the model never learned how a spec conditions output. A prompt selects a conditional behavior the model already learned; it cannot install one it didn't. The same applies to RAG and tool outputs: they extend the conditioning sequence, but only condition effectively if the model learned how such materials constrain continuations.

The honest caveats

This is a pure theory paper. No experiments. Its persuasiveness comes from logical rigor, not data. It derives the conditions that *should* hold, not that they do hold in real LLMs.
The author is an independent researcher — no institutional peer-pressure testing.
No model-scale distinction. Whether larger models indirectly overcome some sufficiency barriers isn't predicted; experiments could overturn the framework.
English-only by default. Multilingual mixture identifiability is not addressed.
No solutions. It is a problem statement — a very precise one. If hallucination is structural information absence, better LLM training alone won't fix it.
The paper's own boundary: it addresses whether models can learn the correct text-conditional distribution, not whether high-probability continuations are *true* — the latter requires fact-checking, symbolic verification, external execution.

Stepping back: a quiet paradigm crack

This paper proposes no new model, no benchmark wins, no 8×H200 runs. It just examines the premise everyone accepts and asks: in what sense, under what conditions, is it correct?

Next-token prediction as the whole of language understanding decomposes into three condition groups:

Learnability: corpus stationarity, ergodicity, mixture identifiability.
Usefulness: the text prefix is a sufficient statistic for the hidden environment.
Verifiability: even if both hold, truth of continuations still requires external checking.

Where they hold are the local sufficiency islands — programming, textbook theorem proving, templated legal documents. Where they fail are open-world factual domains — historical causation, medical diagnosis, political judgment.

The paper ultimately delivers not a solution but a set of criteria: when using next-token prediction as language understanding, check how large $I(X_{t+1}; Z_t \mid X_{\leq t})$ actually is in your domain.

References

1. Corielli, F. (2026). When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming. *arXiv:2605.23278*. 2. Bender, E.M. & Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. *ACL 2020*. 3. Shumailov, I., et al. (2024). The Curse of Recursion: Training on Generated Data Makes Models Forget. *arXiv:2305.17493*. 4. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. *NeurIPS 2020*. 5. Shannon, C.E. (1948). A Mathematical Theory of Communication. *Bell System Technical Journal*.