Summary
This paper, presented on zhichai.net, addresses reliability problems in measuring bias in Large Language Models. Common evaluation methods compare the log-likelihoods of two contrastive stereotype sentences, but the authors argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. The authors propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. They contribute a data-augmentation framework that fills critical gaps in existing stereotype datasets by generating paraphrases and alternate attributes, applied to English, Russian, Spanish, and Chinese stereotypes. They also introduce two evaluation metrics tailored to the dual minimal pair setup, one of which models the mutual information between social groups and stereotype attributes, offering a new perspective on bias. The MI-based metric is better suited to aggregation and enables more robust comparisons of stereotype strength across different languages and models. Authors: Nataliya Stepanova, Ivan Titov, Emily Allaway, Björn Ross (arXiv: 2610.08747, published 2026-10-06).
Paper Overview
- Field: NLP
- Authors: Nataliya Stepanova, Ivan Titov, Emily Allaway, Björn Ross
- Published: 2026-10-06
- arXiv: 2610.08747
Summary
A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation.
First, we present a data-augmentation framework that fills critical gaps in existing stereotype datasets by generating paraphrases and alternate attributes. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes.
Second, we introduce two evaluation metrics tailored to the dual minimal pair setup. One of these metrics provides a new perspective on bias by modeling the mutual information (MI) between social groups and stereotype attributes. This MI-based metric is better suited to aggregation and enables more robust comparisons of stereotype strength across different languages and models.
---
*Auto-collected on 2026-10-08.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635729