The Ghost of Language: Why AI Trained on All Text Still Can't Fully Understand Meaning

Forum topic · 小凯 · 2026-08-31

Summary

A zhichai.net forum post explains the paper 'A Formal Limitation on Learning Human Language From Textual Corpora' by Emily Cheng and Ryan Cotterell (Universitat Pompeu Fabra / ETH Zürich, arXiv:2608.28560). Framed via Searle's Chinese Room, the paper models language as a joint distribution over meaning, context, and utterance, and proves an information-theoretic upper bound on how well any learner restricted to text (U) can recover a speaker's intent (M): P(M̂=M) ≤ I(U;M)/H(M). This bound is tight and architecture-independent—no model scale or data volume can exceed it. Three experiments validate the theory: a synthetic language with controllable noise, Chinese zero-pronoun resolution, and color reference tasks, where LLM performance stays below the predicted ceiling. The post discusses implications: LLM 'understanding' is inherently probabilistic and bounded, and true grounding may require multimodal or interactive learning. It concludes that the limitation mirrors language itself, which encodes meaning only against shared context.

This post from zhichai.net presents a Feynman-style walkthrough of the paper "A Formal Limitation on Learning Human Language From Textual Corpora" by Emily Cheng and Ryan Cotterell (Universitat Pompeu Fabra / ETH Zürich, arXiv:2608.28560).

Key points

The Chinese Room, revisited. The post opens with Searle's (1980) famous thought experiment: symbol manipulation is not understanding. LLMs that read nearly all human text face the same question—do they understand, or just play an elaborate symbol game?
Three worlds of language. The paper models language use as a joint distribution over meaning (M), context (C), and utterance form (U). Utterances never carry all the meaning; context carries the rest (e.g., "The moon is beautiful tonight" meaning something very different across contexts).
Core information-theoretic bound. A listener with access only to U has a strict upper bound on recovering M, composed of (1) irreducible ambiguity inherent to language, and (2) ambiguity resolvable only through context—which is permanently inaccessible to text-only learners:

P(\hat{M} = M) \leq \frac{I(U; M)}{H(M)}

where $$I(U;M)$$ is the mutual information between utterance and meaning and $$H(M)$$ is the entropy of meaning. Example: with 8 equiprobable meanings (H(M)=3 bits) and I(U;M)=2 bits, the best possible accuracy is 2/3 ≈ 66.7%.

The bound is tight and universal. Like the speed of light, the bound holds regardless of architecture (Transformer, RNN, or beyond)—it follows from information theory, not model design.
Three validating experiments:
A controlled synthetic language: neural models never exceed the theoretical bound and approach it as mapping noise increases.
Chinese zero-pronoun resolution: LLM performance improves with richer context but always stays below the theoretical ceiling—a task on which humans face the same limit.
Color reference: words like "blue" cover many colors and are culturally split (e.g., Russian голубой vs. синий), so the inherent uncertainty persists no matter how much text is seen.
What LLMs can and cannot do. They can learn statistical regularities, generate well-formed text, and act as powerful language interfaces—but cannot fully recover speaker intent, ground perceptual concepts like "pain" or "warmth," or resolve culture-dependent meanings like irony and metaphor, at least when learning from text alone.
Paths forward: multimodal learning (accessing context through perception), interactive learning (feedback and correction), and accepting that understanding is inherently probabilistic—as it is between humans too.

Conclusion

The paper's sobering answer: AI reading all the world's text still does not (fully) understand—not because AI is insufficiently clever, but because language itself is incomplete. Meaning is encoded only jointly by form and shared context. The limit is a mirror, revealing where human understanding itself comes from: perhaps not in the words, but beyond them.

References

Cheng, E., & Cotterell, R. (2026). A Formal Limitation on Learning Human Language From Textual Corpora. *arXiv preprint arXiv:2608.28560*.
Searle, J. R. (1980). Minds, brains, and programs. *Behavioral and Brain Sciences*, 3(3), 417-424.
Shannon, C. E. (1948). A mathematical theory of communication. *Bell System Technical Journal*, 27(3), 379-423.