WinoGrande: An Adversarial Winograd Schema Challenge at Scale

Overview

WinoGrande is a large-scale benchmark for commonsense reasoning, presented in the paper *"WinoGrande: An Adversarial Winograd Schema Challenge at Scale"* by Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. The paper is available on arXiv at https://arxiv.org/abs/1907.10641.

The dataset modernizes the classic Winograd Schema Challenge, a pronoun-resolution task that tests whether a system can resolve ambiguous references using commonsense knowledge rather than shallow statistical cues.

Motivation

Existing Winograd-style datasets are small and contain annotation artifacts: models can achieve high accuracy by exploiting spurious lexical or statistical patterns instead of reasoning. WinoGrande addresses both problems — scale and adversarial filtering.

Approach

1. Crowdsourced collection at scale: 44,000 WinoGrande-style problems are gathered via a crowdsourcing protocol with novel validation mechanisms to control quality. 2. AFLITE (Adversarial Filtering): an algorithm that iteratively removes examples that are easy for statistical models (embedding-based classifiers), retaining only instances that require genuine commonsense reasoning. 3. Resulting benchmarks: WinoGrandeXL (12,282 validated examples) and a held-out test set of 1,767 problems.

Key Findings

• Models that perform well on the original small Winograd Schema Challenge datasets fail to transfer to WinoGrande, confirming that prior success relied on dataset-specific biases.
• State-of-the-art models at the time of publication underperform humans on WinoGrande by a large margin, establishing it as a challenging benchmark.
• WinoGrande supports transfer learning: training on it improves performance on related commonsense reasoning datasets.

Resources

• Paper: https://arxiv.org/abs/1907.10641
• Dataset and leaderboard are publicly released by the authors.

Relevance

WinoGrande is listed under the *Evaluation* section of this collection as a standard benchmark for measuring commonsense reasoning and pronoun resolution in language models. It pairs well with other evaluation resources on retrieval-augmented generation and multi-hop QA, forming part of a broader evaluation pipeline for modern LLMs.