SciQ: Crowdsourcing Multiple Choice Science Questions (Welbl, Liu, Gardner, 2017)

Forum topic · 小凯 · 2026-07-05

Summary

SciQ is a crowdsourced dataset of 13,679 multiple-choice science questions introduced by Johannes Welbl, Nelson F. Liu, and Matt Gardner (Allen Institute for Artificial Intelligence / University of Washington) in the paper "Crowdsourcing Multiple Choice Science Questions" (arXiv:1707.06209, 2017). To build the dataset, crowd workers were shown a related science paragraph and an answer from an existing exam question set, then asked to write a novel question; distractors were gathered either from crowdworkers in a gamified setting or via automatic collection methods. Questions cover physics, chemistry, biology, and other middle-school-level science topics. The authors show that crowdworkers can produce high-quality exam-style questions at scale, and that neural models trained on SciQ approach state-of-the-art performance on the AI2 Science Questions dataset without task-specific engineering. SciQ remains a widely used benchmark for evaluating reading comprehension and knowledge-based question answering models. This post summarizes the dataset's construction methodology, quality controls, baseline experiments, and its relevance for evaluation of retrieval and question-answering systems.

SciQ: Crowdsourcing Multiple Choice Science Questions (arXiv:1707.06209)

Overview

| Field | Content | |-------|---------| | Paper | Crowdsourcing Multiple Choice Science Questions | | Authors | Johannes Welbl, Nelson F. Liu, Matt Gardner (Allen Institute for Artificial Intelligence; University of Washington) | | Year | 2017 | | Link | https://arxiv.org/abs/1707.06209 | | Resource | Academic paper + public dataset (SciQ) | | Section | Evaluation of Search Engines / QA benchmarks |

Key Points

Problem: Existing science QA datasets (e.g., AI2 Science Questions) are small and expensive to create, limiting the development of models that need broad science knowledge.
Approach: The authors crowdsource new exam-style multiple-choice questions. Workers are shown a supporting science paragraph and a candidate answer, and asked to write a question whose answer is the given one — this grounds questions in real science content.
Distractor collection: Wrong answer options are gathered in two ways: (1) from crowdworkers, framed as a gamified task of generating plausible-but-wrong answers, and (2) automatically, using answer-similarity methods over a large text corpus.
Dataset: SciQ contains 13,679 questions covering physics, chemistry, biology, and related middle-school science topics, split into train/dev/test, each with an answer, three distractors, and (for many items) supporting evidence text.
Quality control: Filtering and human validation are applied to remove ambiguous, trivial, or incorrectly answered questions; worker agreement checks keep quality comparable to expert-written exam questions.
Experiments: Simple neural models (e.g., a max-entropy model and distance-based ranking neural network using embedding features) trained on SciQ reach performance near state-of-the-art on the AI2 Science Questions without task-specific engineering, demonstrating transfer value of the crowdsourced data.
Takeaway: Crowdsourcing, with answer-anchored question writing and careful distractor sourcing, can scale up domain-specific MCQA datasets that would otherwise require domain experts.

Why It Matters

SciQ became a standard benchmark for reading comprehension and knowledge-based QA, later appearing in evaluation suites for LLMs and retrieval-augmented systems.
The answer-conditioned question generation recipe and the crowd vs. automatic distractor comparison are reusable patterns for building new datasets.
It complements related resources such as the AI2 Science Questions and Natural Questions-style QA sets (see cross-references below).

Cross-References

Evaluation of Retrieval-Augmented Generation: A Survey
A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts
ARES: An Automated Evaluation Framework for RAG

> Note: Quantitative results above follow the published paper; consult the PDF for exact tables.

Reference: Welbl, J., Liu, N. F., & Gardner, M. (2017). *Crowdsourcing Multiple Choice Science Questions*. arXiv:1707.06209.