A Formal Limitation on Learning Human Language From Textual Corpora (Cheng & Cotterell, arXiv 2608.28560)

Forum topic · 小凯 · 2026-09-01

Summary

This arXiv paper (2608.28560) by Emily Cheng and Ryan Cotterell asks whether a listener can recover a speaker's meaning from the form of an utterance alone. The authors take an information-theoretic approach that applies to any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, they derive upper bounds on the probability that a decoder recovers the speaker's intended meaning from an utterance representation. The bounds are governed by the uncertainty that linguistic form leaves about meaning, which decomposes into an irreducible part and a part resolvable only by extralinguistic context, never by the utterance alone. Because these quantities are intrinsic properties of language, no representation—regardless of the amount of text or supervision used to produce it—can surpass them; the bounds hold for both discrete and continuous meaning spaces. Experiments on artificial languages, Chinese zero-pronoun resolution, and color reference empirically support the theory. The result establishes a fundamental limitation on learning human language from textual corpora alone.

Paper Overview

Field: NLP
Authors: Emily Cheng, Ryan Cotterell
Posted: 2026-08-28
arXiv: 2608.28560

Abstract (translated)

Can a listener recover what a speaker means from the form of an utterance alone? This paper answers the question information-theoretically, for a listener given by any featurizer of text—including the hidden states of contemporary large language models.

Modeling language use as a joint distribution over meanings, contexts, and utterances, the authors derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance. The bounds are governed by the uncertainty that form leaves about meaning, which splits into:

1. An irreducible part, and 2. A part that only extralinguistic context—never the utterance alone—can resolve.

Because these quantities are intrinsic to language, no representation, however much text or supervision produced it, can surpass them. The bounds hold whether the meaning space is discrete or continuous.

Experiments on artificial languages, Chinese zero-pronoun resolution, and color reference provide empirical evidence for the theory.

Original Abstract (excerpt)

> Can a listener recover what a speaker means from the form of an utterance alone? We answer this question information-theoretically, and for a listener given by any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, we derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance. The bounds are governed by the uncertainty that form leaves about meaning...

---

*Auto-collected on 2026-09-01*