WinoGrande: An Adversarial Winograd Schema Challenge at Scale

This entry indexes the WinoGrande dataset and paper by Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi (Allen Institute for AI). Paper link: https://arxiv.org/abs/1907.10641.

Key points

• What it is: A large-scale collection of Winograd Schema–style problems — twin sentences that differ by one or two words and require commonsense reasoning to resolve an ambiguous pronoun.
• Scale: Approximately 44,000 crowd-sourced problems, an order of magnitude larger than prior Winograd-style datasets, divided into five training sets of increasing size (xs, s, m, l, xl) for studying data scaling.
• Adversarial filtering (AFLITE): A novel algorithm that identifies and filters out examples that crowdworkers (and biased models) can solve using spurious lexical/association cues, yielding a debiased, challenging test and dev sets.
• Result: State-of-the-art pretrained models (e.g., RoBERTa) trained on the full set still trail human performance, demonstrating that WinoGrande serves as a robust unsolved benchmark for commonsense reasoning. Human performance is reported at ~94%, while the best models at publication were notably lower.
• Transfer: Models pretrained on WinoGrande transfer better to other commonsense reasoning datasets, suggesting its value beyond standalone evaluation.

Context within this collection

The entry is filed under Evaluation of Search engines and sits alongside related resources:

• Evaluation of Retrieval-Augmented Generation: A Survey
• ARES: An Automated Evaluation Framework for RAG
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
• AI Search Has A Citation Problem (CJR, Mar 2025)

Why it matters for IR/LLM evaluation

Benchmarks like WinoGrande address a recurring problem in evaluation: annotation artifacts let models score highly without genuine reasoning. AFLITE-style adversarial filtering is directly relevant to building trustworthy retrieval and QA benchmarks, where shortcut learning can inflate offline metrics that do not translate to real user satisfaction.

Notes and caveats

• This post is a metadata summary with contextual analysis; quantitative results should be verified against the original paper PDF.
• The dataset and leaderboard are maintained by the Allen Institute for AI.

Glossary

| Term | Meaning | |------|---------| | Winograd Schema | Sentence pair requiring commonsense reasoning to disambiguate a pronoun | | AFLITE | Adversarial filtering algorithm that removes bias-exploitable examples | | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric |

Reference

• Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. *WinoGrande: An Adversarial Winograd Schema Challenge at Scale*. arXiv:1907.10641. https://arxiv.org/abs/1907.10641