Language Cannot Be Learned From Text Alone: An Information-Theoretic Proof

A Baby's Dilemma

Imagine a baby who, from birth, hears countless sentences—"pass me the cup," "nice weather today," "are you hungry"—but never sees anyone's intent, never witnesses the situations in which speech occurs, and receives only a pure stream of text.

Can this baby learn language? This is not a thought experiment; it is the real situation of every large language model. GPT, Claude, and Llama are all this baby—except they hear trillions of tokens rather than a few years of conversation.

In August 2026, Emily Cheng (Pompeu Fabra University) and Ryan Cotterell (ETH Zurich) published a paper on arXiv offering a stark information-theoretic answer: no. No matter how much text, how large a model, or how good a training algorithm you use, there is an insurmountable upper bound on the probability of recovering speaker intent from pure text.

This bound is not an engineering limitation—it is a mathematical one.

A Communication Model: Speaker, Listener, and Invisible Intent

The paper builds a communication model with three roles:

• Speaker: holds an intent (meaning M), and in a context (C) chooses an utterance (U)
• Listener: hears U and tries to recover M from it
• Context: all information shared by speaker and listener beyond the utterance itself

The key point: the utterance U is a single sample from the joint distribution of intent M and context C. "Pass me that" refers to a cup in the kitchen and a wrench in the workshop. The utterance alone does not carry enough information to distinguish these intents.

This is not a problem of "existing models being insufficient"—it is a property of language itself.

Two Theorems: An Insurmountable Upper Bound

Theorem 1 (discrete meaning spaces): For any mapping g from utterances to representations (including the internal activations of a perfect learner), a decoder f's probability of correctly recovering intent is bounded by:

\[p_e \leq \frac{I(M; U)}{H(M)} + \text{context-resolvable part}\]

where I(M;U) is the mutual information between intent and utterance, and H(M) is the entropy of intent.

Theorem 2 (continuous meaning spaces): An analogous upper bound holds in continuous spaces.

The bounds are governed by two quantities:

1. The irreducible part: uncertainty about intent that the utterance itself cannot eliminate—intrinsic to the structure of language, beyond any textual representation. 2. The context-resolvable part: uncertainty that only context (not the utterance) can eliminate.

The crucial insight: context is indispensable for language understanding, but context is not in the text. You can train a model on trillions of tokens, but as long as the input contains only the utterance U and not the context C, the bound holds.

An Analogy: Cryptography and Translation

Imagine deciphering a ciphertext whose length is limited while the plaintext comes from an unbounded message space. Even with perfect cryptanalysis tools, if the ciphertext is too short, you can never determine the plaintext—multiple plaintexts map to the same ciphertext.

Language works the same way. An utterance is a "ciphertext" of intent, but with information loss. Different intents can produce the same utterance: "I'm fine" can be sincere or a conversation-ending deflection. Without context, you cannot distinguish them.

This shares the spirit of Shannon's channel capacity theorem: information lost in transmission cannot be recovered downstream. Language's "channel" (the utterance) inherently loses information, because speakers must compress unbounded intent into finite symbols.

Experiments: Verifying the Bound in Three Language Tasks

The authors validate the bound empirically:

1. Artificial languages

Discrete and continuous artificial languages were constructed with precisely controlled I(M;U) and H(M). Decoder recovery probabilities strictly followed the theoretical bounds—whether using MLPs or LLMs as feature extractors, the bound could not be broken.

2. Mandarin pro-drop resolution

In Chinese, "吃了" ("ate") can mean "I ate," "you ate," or "he ate"—pronouns are dropped and intent depends on context. Models must infer the elided pronoun from the utterance alone. Accuracy was tightly constrained by I(M;U)/H(M), matching theoretical predictions.

3. Color naming

In Monroe et al. (2017)'s color naming task, a speaker sees a color and describes it with a word; a listener must recover the color from the word. The meaning space is continuous; the utterance is discrete. Again, model recovery error was constrained by the bound.

Three experiments—from artificial to natural, from discrete to continuous—all confirmed the bound.

What This Means

1. LLM "understanding" is statistical approximation, not semantic recovery

LLMs are trained on pure text. By Theorems 1 and 2, their internal representations cannot break the information-theoretic bound: they learn the statistical structure of utterance distributions, not intent. This doesn't make LLMs useless—statistical structure is rich enough for translation, summarization, and QA—but it means LLM understanding is fundamentally different from human understanding, where discourse uses both utterances and context.

2. Multimodality is not an optional optimization; it is a mathematical necessity

If pure text cannot break the bound, the only way forward is injecting context. Vision, audio, and embodied signals are mathematical necessities, not refinements—which explains why multimodal capabilities in GPT-4V and Gemini are more than decorative.

3. RAG is not retrieval augmentation; it is context injection

RAG's essence is not "looking things up" but injecting context C. By placing external knowledge into the prompt, RAG shrinks H(M|U) and enlarges I(M;U), raising the recoverable probability. From an information-theoretic view, RAG is the mathematically necessary channel for breaking the pure-text bound.

4. A mathematical root of alignment

Why do LLMs hallucinate? Why is alignment hard? The paper offers a mathematical root: LLM internal representations inherently lack complete intent information. Alignment is not making a model "tell the truth"—it is making a model behave reasonably despite incomplete information.

Connections to Existing Conceptual Lineages

• Evaluation blind spots: we evaluate LLM "understanding" via text, but text does not carry full intent. Evaluation measures utterance-distribution matching, not intent recovery—separated by an information-theoretic bound.
• Judgment–output decoupling: an LLM may internally "know" the correct answer while being unable to recover full intent in output, because the bound limits recoverability.
• Solving problems at a different level: pure text cannot break the bound, but multimodality, RAG, and embodied interaction can—by introducing new information sources, not by "trying harder" at the text level.

An Honest Assessment

The paper has limitations:

• Theory: the bounds are existential and give no concrete values. We know the bound exists but not how tight it is for natural language—if loose (e.g., 0.99), the practical constraint is minor; if tight (e.g., 0.5), LLM "understanding" would forever be at most half.
• Experiments: all three tasks are controlled/artificial-semi-natural. Actual I(M;U) and H(M) for natural language are extremely hard to estimate, and the paper gives no actual bound values for natural language.
• Meaning: the result guides LLM engineering (multimodality, RAG are necessary directions) but does not settle the philosophical debate about whether LLMs truly understand language—it only says "pure text is insufficient," not "what would suffice."

Still, as an information-theoretic proof, its value lies in transforming a vague philosophical question ("do LLMs really understand language?") into a precise mathematical one ("what is the upper bound on recovering M from U?"). That conversion itself is a contribution.

Conclusion

Half a century ago, Gold (1967) proved a negative result: the correct grammar cannot be guaranteed to be learned from a stream of positive text examples. Half a century later, Cheng and Cotterell give the information-theoretic version: intent cannot be guaranteed to be recovered from pure text.

Neither result says "LLMs are useless." They say something more precise: text is a lossy projection of intent, and no projector, however good, can reconstruct the original from its projection.

Understanding this explains why LLMs need RAG, multimodality, and tool calling—not as engineering optimizations, but as mathematical necessities.

---

Paper: A Formal Limitation on Learning Human Language From Textual Corpora Authors: Emily Cheng (UPF), Ryan Cotterell (ETH Zürich) Date: August 28, 2026