BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019)

Overview

This entry indexes the paper "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions" (arXiv:1905.10044, 2019), listed under the *Evaluation of Search Engines* section of the collection.

• Authors: Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova
• Link: https://arxiv.org/abs/1905.10044
• Type: Academic paper / dataset release

What BoolQ Is

BoolQ is a benchmark dataset for yes/no (boolean) question answering. Its distinguishing feature is that the questions are naturally occurring: they are harvested from real user search queries and paired with Wikipedia passages that contain the answer. This contrasts with prior reading-comprehension datasets whose questions were written by annotators and tend to be more formulaic.

The paper's title reflects its central finding: yes/no questions posed by real users are surprisingly difficult for reading-comprehension models, because they require nuanced inference over evidence rather than simple span extraction.

Key points

• BoolQ consists of naturally asked yes/no questions linked to Wikipedia paragraphs containing the evidence needed to answer them.
• Because the questions are unconstrained and drawn from real queries, they exhibit far more linguistic diversity than crowdsourced benchmarks.
• The paper shows a strong transfer learning effect: pre-training on synthetic, automatically generated yes/no questions substantially boosts BERT performance before fine-tuning on BoolQ.
• The dataset became a standard QA benchmark and is commonly used in evaluating retrieval and question-answering systems.

Context in This Collection

The entry is cross-referenced with related items on evaluation and retrieval, including:

• Evaluation of Retrieval-Augmented Generation: A Survey
• ARES: An Automated Evaluation Framework for RAG
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles
• AI Search Has A Citation Problem (CJR, 2025)

Caveats

This post is largely a metadata template; detailed numerical results, exact dataset size, and baseline scores should be verified against the original paper PDF before citing quantitative conclusions.

References

• Original paper: BoolQ on arXiv