OpenBookQA: A New Dataset for Open Book Question Answering (AllenAI, 2018)

Forum topic · 小凯 · 2026-07-05

Summary

OpenBookQA is a question answering dataset introduced by Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal at Allen Institute for AI (arXiv:1809.02789, September 2018, presented at EMNLP 2018). Inspired by open book exams, it models a scenario in which a reader has broad access to a small 'book' of 1,326 core science facts plus additional knowledge, and must combine the relevant fact with broad common knowledge to answer. The dataset contains 5,957 four-way multiple-choice questions. Each question is associated with one core fact, but answering typically requires additional background knowledge—for example, knowing that a suit of armor is made of metal, and that metal conducts electricity. The paper shows that both human readers and neural models rely on this complementary knowledge: human crowdworkers achieve about 92% accuracy, while state-of-the-art preprocessed QA methods of the time scored well below that, and simple fact-matching baselines perform poorly, confirming that the dataset probes deeper reasoning rather than surface retrieval. OpenBookQA became a standard benchmark for testing knowledge-grounded reasoning in NLP systems.

OpenBookQA: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Authors: Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal (Allen Institute for Artificial Intelligence)
Published: September 2018 (arXiv:1809.02789; EMNLP 2018)
Link: https://arxiv.org/abs/1809.02789

Abstract (original)

> We introduce a new question answering dataset, OpenBookQA, which requires combining a core science fact with broad common knowledge to answer. The questions are modeled after open book exams: the reader has access to a small set of facts but must apply additional knowledge to reach the correct answer. The dataset contains 5,957 multiple-choice questions.

Key Points

Motivation: Model question answering after *open book exams* — a human with access to a small set of facts can answer questions whose answers are not literally stated in those facts, because humans bring extensive extra knowledge to bear.
Structure: Each question is paired with one of 1,326 core science facts (the "book"). Answering requires combining the retrieved fact with broad common knowledge.
Scale: 5,957 four-way multiple-choice questions, crowdsourced and filtered to require additional knowledge beyond the core fact.

Example

The title question — *Can a suit of armor conduct electricity?* — is answerable by combining the core fact "metal conducts electricity" with the common knowledge that suits of armor are made of metal. Neither piece alone is sufficient.

Baselines and Findings

The paper evaluates simple retrieval-style baselines, neural models, and human annotators.
Human performance is high (~92%), while models of the time lag substantially behind, especially those without access to external knowledge.
A large fraction of questions cannot be answered by the core fact alone, confirming the dataset probes reasoning and knowledge integration rather than fact lookup.
Providing the correct core fact to models improves accuracy, and the paper analyzes where models fail (e.g., needing additional lexical or conceptual knowledge).

Significance

OpenBookQA became a widely used benchmark for knowledge-grounded and commonsense reasoning in NLP, complementing datasets such as ARC and ScienceQA. It highlighted that even with an "open book," answering science questions requires broad background knowledge — a precursor to modern retrieval-augmented generation (RAG) evaluations.

Limitations

Focused on elementary-level science; domain coverage is narrow.
Multiple-choice format simplifies evaluation compared with open-ended generation.
Crowdsourced questions may contain noise, and later large language models approach ceiling performance on this benchmark.

References

Original paper: https://arxiv.org/abs/1809.02789