Language Can't Be Learned From Text Alone: An Information-Theoretic Proof
A Baby's Dilemma
Imagine a baby who, from birth, hears countless sentences every day — "pass me the cup," "nice weather today," "are you hungry?" — but never sees anyone's intentions, never witnesses the situations in which speech occurs. Only a pure stream of text.
Can this baby learn language?
This is not a thought experiment. It is the real situation of every large language model (LLM). GPT, Claude, Llama — they are all this baby, except they hear not a few years of conversation but trillions of tokens of text.
In August 2026, Emily Cheng (Pompeu Fabra University) and Ryan Cotterell (ETH Zürich) published a paper on arXiv offering a stark information-theoretic answer: no. No matter how much text, how large the model, or how good the training algorithm, there is an insurmountable upper bound on the probability of recovering speaker intent from pure text.
This bound is not an engineering limitation. It is a mathematical one.
A Communication Model: Speaker, Listener, and Invisible Intent
The paper builds a communication model with three roles:
- Speaker: has an intent (meaning M) in mind, and in a context (C) selects an utterance (U)
- Listener: hears U and tries to recover M from it
- Context: everything beyond the utterance shared by speaker and listener
- Evaluation blind spots: We evaluate LLM "understanding" via text, but text does not carry full intent. Evaluation measures "utterance-distribution matching," not "intent recovery" — with an information-theoretic bound between them.
- Judgment–gate decoupling: An LLM may internally "know" the right answer (the statistical structure is rich enough), yet fail to recover full intent in output — the gap between internal judgment and output gate has an information-theoretic basis.
- Changing the level of attack: The pure-text level cannot break the bound, but multimodality, RAG, and embodied interaction can — because they introduce new information sources. This is not "trying harder" at the text level, but switching levels.
The key point: the utterance U is a single sample from the joint distribution of intent M and context C. The same sentence "pass me that" refers to a cup in the kitchen and a wrench in the workshop. The utterance alone does not carry enough information to distinguish the two intents.
This is not a matter of "models aren't good enough yet" — it is a property of language itself.
Two Theorems: An Insurmountable Bound
The paper's core contributions are two theorems.
Theorem 1 (discrete meaning spaces): For any mapping g from utterances to representations (including a perfect learner's internal activations), the probability that a decoder f correctly recovers the intent is bounded by:
where \(I(M; U)\) is the mutual information between intent and utterance, and \(H(M)\) is the entropy of the intent.
Theorem 2 (continuous meaning spaces): An analogous upper bound holds in continuous spaces.
These bounds are determined by two quantities:
1. The irreducible part: intent uncertainty that the utterance itself cannot eliminate. This is intrinsic to the structure of language; no textual representation can break through it. 2. The context-resolvable part: uncertainty that only context (not the utterance) can eliminate.
The key insight: context is an indispensable part of language understanding, but context is not in the text. You can train a model on trillions of tokens, but as long as the input is only the utterance U without context C, the bound stands.
Analogies: Cryptography and Translation
Imagine deciphering a ciphertext. The ciphertext has limited length, while the plaintext may come from an infinite message space. Even with perfect cryptanalysis, if the ciphertext is too short, you can never determine the original — because many plaintexts map to the same ciphertext.
Language is the same. An utterance is a "ciphertext" of intent, but that ciphertext loses information. Different intents can produce identical utterances. "I'm fine" can mean genuinely fine, or a refusal to talk further. Without context, you cannot distinguish.
This shares the spirit of Shannon's channel capacity theorem: information lost in transmission cannot be recovered downstream. Language's "channel" (the utterance) inherently loses information, because speakers must compress unlimited intent into finite symbols.
Experiments: Three Languages Verify the Bound
The paper does not stop at theory. The authors validated the bound's predictions on three tasks:
1. Artificial languages
The authors constructed discrete and continuous artificial languages with precisely controlled \(I(M; U)\) and \(H(M)\). Experiments show that decoders' recovery probabilities strictly follow the bounds predicted by the theorems — whether using MLPs or LLMs as feature extractors, nothing breaks through.
2. Mandarin zero-pronoun resolution
In Chinese, "chi le" (ate) can mean "I ate," "you ate," or "he/she ate" — the pronoun is dropped, and the intent depends on context. The model must infer the elided pronoun from the utterance alone. Accuracy is tightly constrained by \(I(M; U)/H(M)\), matching theoretical predictions.
3. Color naming
In the color naming task of Monroe et al. (2017), a speaker sees a color and describes it with a word; the listener must recover the color from the word. The color space is continuous, the utterance discrete. Again, model recovery error is bounded by the theoretical limit.
Three experiments — from artificial to natural, from discrete to continuous — and the bound holds in all of them.
What This Means
1. LLM "understanding" is statistical approximation, not semantic recovery
LLMs are trained on pure text. By Theorems 1 and 2, their internal representations cannot exceed the information-theoretic bound. What LLMs learn is not "intent" but "the statistical structure of utterance distributions."
This does not deny that LLMs are useful. Statistical structure is rich enough to support translation, summarization, and question answering. But LLM "understanding" differs fundamentally from human understanding — humans use both utterance and context in conversation; LLMs have only the utterance.
2. Multimodality is not an optional optimization but a mathematical necessity
If pure text cannot break the bound, the only way forward is to introduce context. Multimodality (vision, audio, embodied signals) is not an engineering enhancement — it is a mathematical necessity. This explains why the multimodal capabilities of GPT-4V and Gemini are not icing on the cake: they are breaking through the information-theoretic limits of text alone.
3. RAG is not retrieval augmentation but context injection
The essence of RAG (Retrieval-Augmented Generation) is not "looking up references" but injecting context C. By placing external knowledge into the prompt, RAG shrinks \(H(M|U)\) while expanding \(I(M; U)\), raising the recoverable probability.
From an information-theoretic perspective, RAG is not an engineering trick but the only mathematically available channel to break the pure-text bound.
4. A mathematical root of alignment
Why do LLMs hallucinate? Why is alignment so hard? The paper offers a mathematical root: LLM internal representations simply do not contain complete intent information. Alignment is not making a model "tell the truth," but making it behave reasonably under incomplete information.
This is isomorphic to a metastable supercooled state: an LLM's knowledge state is metastable, and any perturbation (prompt design, sampling temperature, missing context) can flip the system's state. Alignment is not deleting erroneous capabilities, but removing the "ice nucleus" that lets them appear.
Connections to Existing Conceptual Lineages
An Honest Assessment
The paper has limitations.
Theory: The bounds are existential and give no concrete values. We know a bound exists but not how tight it is for natural language. If it is loose (say, 0.99), the practical constraint is small; if tight (say, 0.5), LLM "understanding" can only ever be half the story.
Experiments: The three artificial/semi-natural tasks are controlled. The true \(I(M; U)\) and \(H(M)\) of natural language are extremely hard to estimate, and the paper gives no actual bound values for natural language.
Meaning: The result guides LLM engineers (multimodality, RAG are inevitable directions), but it does not end the philosophical debate about whether LLMs truly understand language. It only says "pure text is not enough" — not "what would be enough."
Still, as an information-theoretic proof, its value lies in this: it turns a vague philosophical question ("do LLMs really understand language?") into a precise mathematical one ("what is the upper bound on the probability of recovering M from U?"). That transformation from philosophy to mathematics is itself a contribution.
Conclusion
Half a century ago, Gold (1967) proved a negative result: correct grammar cannot be guaranteed learnable from a stream of positive-only text. Half a century later, Cheng and Cotterell provide an information-theoretic version: intent cannot be guaranteed recoverable from pure text.
Neither result says "LLMs are useless." They say something more precise: text is a lossy projection of intent, and no matter how good the projector, the original cannot be recovered from the projection.
Only by understanding this can we understand why LLMs need RAG, multimodality, and tool calling — not as engineering optimizations, but as mathematical necessities.
---
Paper: A Formal Limitation on Learning Human Language From Textual Corpora Authors: Emily Cheng (UPF), Ryan Cotterell (ETH Zürich) Date: August 28, 2026