A Formal Limitation on Learning Human Language From Textual Corpora (Cheng & Cotterell, arXiv 2608.28560)

Forum topic · 小凯 · 2026-09-01

Summary

This paper by Emily Cheng and Ryan Cotterell (arXiv:2608.28560) asks whether a listener can recover a speaker's meaning from the form of an utterance alone, answering information-theoretically for any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, the authors derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from an utterance representation. The bounds are governed by the uncertainty that linguistic form leaves about meaning, which splits into an irreducible part and a part resolvable only by extralinguistic context, never by the utterance alone. Because these quantities are intrinsic properties of a language, no representation—regardless of how much text or supervision produced it—can exceed them, and the bounds hold whether the meaning space is discrete or continuous. Experiments on artificial languages, Chinese zero-pronoun resolution, and color reference provide empirical support for the theory. The result formalizes a fundamental limitation of learning human language from textual corpora alone.

Paper Overview

Field: NLP Authors: Emily Cheng, Ryan Cotterell Published: 2026-08-28 arXiv: 2608.28560

Summary

Can a listener recover what a speaker means from the form of an utterance alone? This paper answers the question information-theoretically, for a listener given by any featurizer of text, including the hidden states of contemporary large language models.

Key Contributions

Models language use as a joint distribution over meanings, contexts, and utterances.
Derives upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance.
The bounds are governed by the uncertainty that form leaves about meaning, which splits into:
an irreducible part, and
a part that only (extralinguistic) context, but never the utterance alone, can resolve.
Because these quantities are intrinsic to the language, no representation—however much text or supervision produced it—can surpass the bounds.
The bounds hold whether the meaning space is discrete or continuous.

Experiments

Experiments on artificial languages, Chinese zero-pronoun resolution, and color reference tasks provide empirical evidence supporting the theory.

Original Abstract (excerpt)

> Can a listener recover what a speaker means from the form of an utterance alone? We answer this question information-theoretically, and for a listener given by any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, we derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance. The bounds are governed by the uncertainty that form leaves about meaning, which splits into an irreducible part and a part that only (extralinguistic) context, but never the utterance alone, can resolve. Because these quantities are intrinsic to language, no representation, however much text or supervision produced it, can s...

---

*Auto-collected on 2026-09-01*

A Formal Limitation on Learning Human Language From Textual Corpora (Cheng & Cotterell, arXiv 2608.28560)

Summary

Paper Overview

Summary

Key Contributions

Experiments

Original Abstract (excerpt)

Tags