Can a Suit of Armor Conduct Electricity? Introducing the OpenBookQA Dataset for Open Book Question Answering

Forum topic · 小凯 · 2026-07-05

Summary

OpenBookQA is a question-answering dataset introduced in September 2018 by Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal at the Allen Institute for AI (AllenAI). The paper, titled 'Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering,' is available on arXiv as 1809.02789. The dataset is designed to study question answering that requires combining a small, explicit 'book' of core scientific knowledge facts with broad common knowledge. Each question is associated with one elementary science fact, but answering correctly demands additional background and common knowledge beyond the fact itself, making the task analogous to an open-book exam. The paper analyzes model behavior on this task and shows that existing pre-trained QA methods struggle, especially in the absence of the accompanying fact, motivating research into models that better leverage knowledge. OpenBookQA has become a widely used benchmark for evaluating knowledge-intensive reasoning in NLP systems.

OpenBookQA: A New Dataset for Open Book Question Answering (arXiv 1809.02789, AllenAI, September 2018)

Overview

Paper: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Authors: Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal (Allen Institute for AI) Published: September 2018 Link: https://arxiv.org/abs/1809.02789

Key points

Introduces OpenBookQA, a new question-answering dataset modeled after open-book exams.
Each question is paired with one core scientific fact from a small "book" of elementary-level science knowledge; the task is to combine that fact with broad common knowledge to answer a 4-way multiple-choice question.
The dataset is designed to probe deeper language understanding and knowledge reasoning than fact-retrieval alone: the accompanying fact is necessary but not sufficient to answer the question.
Experiments show that existing pre-trained question-answering methods perform poorly on the dataset, particularly when the associated fact is withheld, indicating significant room for improvement in knowledge-aware reasoning models.
The dataset and resources were released by AllenAI and have since become a standard benchmark for evaluating knowledge-intensive question answering and commonsense reasoning.

Significance

OpenBookQA shifted the evaluation focus from retrieving surface-level facts to combining an explicit knowledge source with common knowledge, making it a useful testbed for measuring how well models reason over knowledge. It complements earlier datasets like ARC and SQuAD by emphasizing inference beyond the provided context.

References

Original paper: https://arxiv.org/abs/1809.02789