Paper Overview
- Paper: A Formal Limitation on Learning Human Language From Textual Corpora
- Authors: Emily Cheng, Ryan Cotterell
- Institutions: Universitat Pompeu Fabra, ETH Zürich
- arXiv: arXiv:2608.28560
- Meaning (M): the speaker's true intent
- Context (C): the situation the utterance occurs in
- Utterance (U): the actual words spoken
- Cheng, E., & Cotterell, R. (2026). A Formal Limitation on Learning Human Language From Textual Corpora. *arXiv preprint arXiv:2608.28560*.
- Searle, J. R. (1980). Minds, brains, and programs. *Behavioral and Brain Sciences*, 3(3), 417-424.
- Shannon, C. E. (1948). A mathematical theory of communication. *Bell System Technical Journal*, 27(3), 379-423.
This post presents a Feynman-style deep dive into the paper, framed around Searle's classic Chinese Room thought experiment: a system that manipulates symbols perfectly may still understand nothing. LLMs like GPT-4, Claude, and Gemini have read nearly all text on the internet—yet do they truly understand it? This paper gives a rigorous, mathematical answer.
Key Points
The Three Worlds of Language
The authors model language as a three-way joint distribution over:
Crucially, form never carries meaning alone. When a friend asks "Want to go in?" at a ramen shop at 2 PM, the context (the shop, the time, the shared hunger) carries most of the information. The same words—Natsume Sōseki's "the moon is beautiful tonight" as a translation of "I love you"—mean different things to different listeners depending on shared cultural background.
The Core Bound
The paper's central theorem: a listener that only ever sees utterances (U) and never context (C) has a strict upper bound on the probability of recovering the speaker's intent:
where \(I(U;M)\) is the mutual information between utterance and meaning, and \(H(M)\) is the entropy of meaning.
The bound has two components:
1. Irreducible ambiguity: inherent in language itself—the same sentence "He saw her duck" could mean seeing her pet bird or seeing her crouch. No amount of compute or data removes this. 2. Context-only-resolvable ambiguity: resolvable in principle by context, but permanently inaccessible to a text-only learner.
The bound is tight: some language systems achieve it, and none can exceed it. Like the speed of light, this is a mathematical necessity—no architecture (Transformer, RNN, or future designs) escapes it. The theorem derives from Shannon's information theory (1948), not from any model's limitations.
Three Experimental Validations
1. Synthetic language: with a controlled noisy mapping from meanings to forms, neural networks never exceeded the theoretical bound, and converged toward it as noise increased. 2. Chinese zero-pronoun resolution: a real natural-language task where subjects are routinely omitted ("Went to the store yesterday, bought apples"). Models performed better with rich context and worse without, but always stayed below the theoretical ceiling—just as humans do. 3. Color reference: "blue" can denote a thousand different blues; LLMs' accuracy in inferring exact colors was constrained by the bound. This raises deeper questions about how language arbitrarily carves continuous perception into categories (Russian has two basic blue terms where English has one).
What This Means for AI
The paper (co-authored by computational linguist Ryan Cotterell) does not dismiss LLMs—it precisely delineates what they can and cannot do.
LLMs can: learn statistical patterns of language, generate grammatical and semantic text, reason over patterns in training data, serve as powerful language interfaces.
LLMs cannot (from text alone): fully eliminate ambiguity to recover true intent; understand concepts requiring physical or perceptual experience (pain, warmth, heaviness); fully grasp culturally dependent meanings (sarcasm, metaphor, puns).
Paths Toward Genuine Understanding
1. Multimodal learning: seeing, hearing, and touching the world gives access to context. But the framework warns: context that remains inaccessible (e.g., a speaker's inner state) still leaves uncertainty. 2. Interactive learning: asking questions, receiving feedback, trial and error—like how children acquire word meanings through correction. 3. Accepting uncertainty: humans also never fully know what another person means; we communicate through sufficiently good guesses. In this light, LLMs' "non-understanding" reflects the nature of language itself rather than a defect.
Conclusion
The paper's answer is sobering: no, AI that reads all the world's text does not (fully) understand it—not because AI is insufficiently intelligent, but because language itself is incomplete. Language was designed to transmit information efficiently among communities sharing context, not to perfectly encode meaning. LLMs are mirrors that reveal the limits of language—and prompt us to ask where human understanding truly comes from. Perhaps the answer lies not in words, but beyond them.