[论文] The Missing Minimal Pair: Stereotype Evaluation in LLMs

论文概要

研究领域: NLP 作者: Nataliya Stepanova, Ivan Titov, Emily Allaway, Björn Ross 发布时间: 2026-10-06 arXiv: 2610.08747

中文摘要

测量大语言模型偏差的常见方法是比较两个对比刻板印象句子的对数似然。我们认为这种单对比较通常不可靠：简单地用替代属性重写相同的刻板印象就可以产生逻辑上不一致的偏好。为此，我们提出了双重最小对设置，为稳健的刻板印象评估引入两个比较轴。首先，我们提出了一个数据增强框架，通过生成改写和替代属性来填补现有刻板印象数据集中的关键空白，并在英语、俄语、西班牙语和中文刻板印象上应用。其次，我们引入了两个针对双重最小对设置的评估指标，其中一个通过建模社会群体和刻板印象属性之间的互信息（MI）为偏差提供了新视角。这种基于MI的指标更适合聚合，并实现了跨不同语言和模型的刻板印象强度的更稳健比较。

原文摘要

A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. First, we present a data-augmentation framework that fills critical gaps in existing stereotype datasets by generating paraphrases and alternate attributes. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes. Second, we introduce two evaluation metrics tailored to the dual minimal pair setup. One of these metrics provides a...

#论文 #arXiv #NLP #小凯