BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, arXiv)

Overview

This entry covers the paper "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions" (arXiv:1905.10044, 2019), authored by Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. The paper is catalogued in this forum under the section Evaluation of Search engines.

• Paper link: https://arxiv.org/abs/1905.10044
• Resource type: Academic paper / dataset

What BoolQ Is

BoolQ is a benchmark dataset for reading comprehension with naturally occurring yes/no questions:

• Questions are real user queries sampled from Google search traffic (anonymized), not crowdsourced or synthetic prompts.
• Each question is paired with a relevant Wikipedia passage and a ground-truth yes/no answer.
• The dataset contains roughly 16k question–passage–answer triples, split into train/dev/test sets.

Why It Is Surprisingly Difficult

Yes/no questions look trivial, but the paper shows they require:

• Understanding entailment and negation within the passage;
• Pragmatic inference (implied rather than explicitly stated facts);
• Handling the ambiguity and imprecision of real-world user queries.

The authors found that a strong pre-trained BERT-based model fine-tuned on BoolQ dramatically outperforms models trained from scratch, while still trailing human accuracy — evidence that natural yes/no questions are harder than their simple format suggests.

Significance for Search & LLM Evaluation

• BoolQ became a standard benchmark in reading-comprehension suites (e.g., SuperGLUE) and later in evaluations of retrieval-augmented generation (RAG) and LLMs.
• It sits at the intersection of information retrieval and question answering: a system must first retrieve the right passage, then make a correct binary decision.

Related Entries in This Collection

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
• A Dataset of Information-Seeking Questions Anchored in Real Web Pages (arXiv:2105.03011)
• ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)

Caveats of This Page

This forum post is a catalog-style entry; quantitative results (exact accuracy figures, baselines) should be verified against the original PDF. The appendix checklists and glossary below are generic notes for readers of the search/recommendation section, not claims from the paper itself.

Reader Checklist

| Concern | Guidance | |---|---| | Reproduction | Use the official BoolQ splits; avoid training on dev/test questions | | Metrics | Report accuracy as in the original paper; add calibration analysis if relevant | | RAG usage | Evaluate retrieval + answer correctness separately |

Glossary

| Term | Meaning | |---|---| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | Entailment | Whether a passage logically supports an answer | | Reading comprehension | Answering questions given a passage |