论文概要
研究领域: NLP 作者: Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim 发布时间: 2026-05-06 arXiv: 2605.05197中文摘要
语法性和似然性是人类语言中不同的概念。预训练语言模型（LM）是拟合最大化语料库似然的概率模型，能够生成语法正确的文本，并在严格控制的最小对中很好地区分语法正确和不正确的句子。然而，它们的字符串概率总体上并不能清晰地区分语法正确和不正确的句子。但LM是否隐式地获得了与字符串概率不同的语法性区分？我们通过研究LM的内部表示来探索这个问题，通过在一个由自然文本语料库应用扰动获得的语法正确和（合成）语法不正确句子的数据集上训练线性探针。我们发现这种简单的语法性探针能够泛化到人工筛选的语法性判断基准，并优于基于LM概率的语法性判断。当应用于语义合理性基准（其中最小对中的两个成员都是语法正确的，仅在合理性上不同）时，探针的表现却不如字符串概率。在英语上训练的探针还表现出非平凡的跨语言泛化能力，在多种其他语言的语法性基准上优于字符串概率。此外，探针分数与字符串概率仅弱相关。这些结果共同表明，LM在其隐藏层中某种程度上获得了隐式的语法性区分。原文摘要
Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitted to maximize corpus likelihood, generate grammatically well-formed text and discriminate well between grammatical and ungrammatical sentences in tightly controlled minimal pairs. However, their string probabilities do not sharply discriminate between grammatical and ungrammatical sentences overall. But do LMs implicitly acquire a grammaticality distinction distinct from string probability? We explore this question through studying internal representations of LMs, by training a linear probe on a dataset of grammatical and (synthetic) ungrammatical sentences obtained by applying perturbations to a naturalistic text corpus. We find that this simple grammaticality probe generalizes to human-curated grammaticality judgment benchmarks and outperforms LM probability-based grammaticality judgments. When applied to semantic plausibility benchmarks, in which both members of a minimal pair are grammatical and differ in only plausibility, the probe however performs worse than string probability. The English-trained probe also exhibits nontrivial cross-lingual generalization, outperforming string probabilities on grammaticality benchmarks in numerous other languages. Additionally, probe scores correlate only weakly with string probabilities. These results collectively suggest that LMs acquire to some extent an implicit grammaticality distinction within their hidden layers.--- *自动采集于 2026-05-08*
#论文 #arXiv #NLP #小凯