Implicit Representations of Grammaticality in Language Models

Forum topic · 小凯 · 2026-05-08

Summary

This paper investigates whether pretrained language models (LMs) implicitly encode a grammaticality distinction that is separable from string probability. The authors train a linear probe on LM internal representations using a dataset of grammatical and synthetically perturbed ungrammatical sentences derived from a naturalistic corpus. The probe generalizes to human-curated grammaticality judgment benchmarks and outperforms grammaticality judgments based on LM string probabilities. Conversely, on semantic plausibility benchmarks—where both sentences in a minimal pair are grammatical—the probe performs worse than string probability, indicating it captures grammatical rather than semantic information. The English-trained probe also shows nontrivial cross-lingual generalization, beating string probabilities on grammaticality benchmarks in numerous other languages. Probe scores correlate only weakly with string probabilities. Together, these findings suggest LMs acquire, to some extent, an implicit grammaticality distinction within their hidden layers, separating grammatical well-formedness from likelihood. Authored by Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, and Yoon Kim; available as arXiv:2605.05197.

Paper Overview

Field: NLP Authors: Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim Published: 2026-05-06 arXiv: 2605.05197

Abstract

Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitted to maximize corpus likelihood, generate grammatically well-formed text and discriminate well between grammatical and ungrammatical sentences in tightly controlled minimal pairs. However, their string probabilities do not sharply discriminate between grammatical and ungrammatical sentences overall. But do LMs implicitly acquire a grammaticality distinction distinct from string probability?

The authors explore this question by studying internal representations of LMs, training a linear probe on a dataset of grammatical and (synthetic) ungrammatical sentences obtained by applying perturbations to a naturalistic text corpus.

Key Findings

Generalization to curated benchmarks: The simple grammaticality probe generalizes to human-curated grammaticality judgment benchmarks and outperforms LM probability-based grammaticality judgments.
Semantic specificity: When applied to semantic plausibility benchmarks, in which both members of a minimal pair are grammatical and differ only in plausibility, the probe performs worse than string probability—suggesting it captures grammar rather than semantics.
Cross-lingual transfer: The English-trained probe exhibits nontrivial cross-lingual generalization, outperforming string probabilities on grammaticality benchmarks in numerous other languages.
Weak correlation with likelihood: Probe scores correlate only weakly with string probabilities.

Conclusion

These results collectively suggest that LMs acquire, to some extent, an implicit grammaticality distinction within their hidden layers—one that is distinguishable from raw string probability.

Original Abstract

> Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitted to maximize corpus likelihood, generate grammatically well-formed text and discriminate well between grammatical and ungrammatical sentences in tightly controlled minimal pairs. However, their string probabilities do not sharply discriminate between grammatical and ungrammatical sentences overall. [...] These results collectively suggest that LMs acquire to some extent an implicit grammaticality distinction within their hidden layers.

---

Paper link: https://arxiv.org/abs/2605.05197