The Ghost of Language: Why AI Reads Everything Yet Still Misunderstands You
This post on zhichai.net is a Feynman-style walkthrough of the paper "A Formal Limitation on Learning Human Language From Textual Corpora" by Emily Cheng and Ryan Cotterell (Universitat Pompeu Fabra and ETH Zürich, arXiv:2608.28560).
The Chinese Room, Revisited
The post opens with Searle's (1980) Chinese Room thought experiment: symbol manipulation does not equal understanding. Modern LLMs like GPT-4, Claude, and Gemini have read nearly all written text and excel at exams, poetry, and reasoning — but do they truly understand? The paper gives a rigorous, mathematical answer.
Three Worlds: Meaning, Context, Utterance
The authors model language use as a three-way joint distribution:
- Meaning (M) — the speaker's true intent ("I'm hungry, want ramen")
- Context (C) — the situation (standing at a ramen shop at 2pm, both unfed)
- Utterance (U) — what is actually said ("Want to go in?")
- Multimodal learning — gaining access to context C via vision/audio, though inner states of speakers remain hidden.
- Interactive learning — narrowing uncertainty through feedback, like children acquiring word meanings by correction.
- Accepting uncertainty — human mutual understanding is also probabilistic and approximate; LLMs' imperfect understanding reflects the nature of language itself, not a defect.
- Cheng, E., & Cotterell, R. (2026). A Formal Limitation on Learning Human Language From Textual Corpora. *arXiv preprint arXiv:2608.28560*.
- Searle, J. R. (1980). Minds, brains, and programs. *Behavioral and Brain Sciences*, 3(3), 417-424.
- Shannon, C. E. (1948). A mathematical theory of communication. *Bell System Technical Journal*, 27(3), 379-423.
Form never carries all the meaning alone; meaning is jointly encoded by form and context. The same sentence — "the moon is beautiful tonight" — means different things to different listeners depending on shared culture and memory (echoing Natsume Sōseki's famous translation of "I love you").
The Core Information-Theoretic Bound
Building on Shannon (1948), the paper proves a listener who sees only U cannot fully recover M:
where \(I(U;M)\) is the mutual information between utterance and meaning, and \(H(M)\) is the entropy of meaning.
Example: with 8 equally likely meanings, H(M) = 3 bits. If I(U;M) = 2 bits, the best possible decoding accuracy is 2/3 ≈ 66.7%. The bound decomposes into:
1. Irreducible ambiguity — inherent in language itself (e.g., "He saw her duck": animal or crouching?). No amount of compute, data, or parameters removes it. 2. Context-resolvable ambiguity — solvable in principle with context C, but permanently inaccessible to a text-only learner. LLMs are like observers who see only ripples on water but never what fell in.
The bound is tight and architecture-independent — it applies to Transformers, RNNs, or any future model, because it follows from information-theoretic first principles, not implementation details. It is a speed-of-light limit on understanding from text.
Three Experiments
1. Synthetic language: With a controlled noisy mapping between meanings and forms, neural networks never exceeded the theoretical bound and approached it as noise increased. 2. Chinese zero-pronoun resolution: A real natural-language task where subjects are routinely omitted ("Went to the store yesterday, bought apples"). LLMs performed better with richer context, but always stayed below the predicted bound — humans face the same limit when context is thin. 3. Color reference: "Blue" can denote countless hues with culturally varying boundaries (Russian distinguishes голубой vs. синий). LLMs matched the theory's constraint on intrinsic referential uncertainty.
What This Means for AI
LLMs can: learn statistical regularities, generate fluent text, reason over training patterns, and serve as language interfaces.
LLMs cannot (from text alone): fully eliminate ambiguity, ground concepts requiring physical or perceptual experience (pain, warmth, weight), or fully grasp culture-bound meaning (irony, metaphor, puns).
Paths forward discussed in the post:
Conclusion
The paper's modest title hides a grand question: when AI has read all the world's text, does it truly understand humanity? The answer: not completely — not because AI is insufficiently clever, but because language itself is incomplete, designed for efficient communication among context-sharing communities, not perfect encoding of meaning. The answer to where human understanding comes from may lie beyond the words.