Crowdsourcing Multiple Choice Science Questions: The SciQ Dataset (arXiv 1707.06209)

Forum topic · 小凯 · 2026-07-05

Summary

SciQ is a crowdsourced multiple-choice science question dataset introduced by Johannes Welbl, Nelson F. Liu, and Matt Gardner (Allen Institute for Artificial Intelligence / University of Washington) in the paper 'Crowdsourcing Multiple Choice Science Questions' (arXiv:1707.06209). The authors show that with careful design, crowdsourcing can produce high-quality science exam-style questions at scale. The dataset combines crowdsourced questions with existing standardized exam questions, pairing each with four answer options: the correct answer and three distractors. Crucially, SciQ provides additional supervision signals beyond the answer itself, including supporting evidence passages and automatic difficulty ratings derived from a question difficulty model, enabling models to leverage distractor selection and question difficulty as training signals. This resource supports research on machine comprehension, question answering, and question generation in the science domain, and has become a widely used benchmark for evaluating reading comprehension models. This page summarizes the paper's motivation, dataset construction methodology, and significance for the NLP and information retrieval community.

Crowdsourcing Multiple Choice Science Questions: The SciQ Dataset

Paper: Crowdsourcing Multiple Choice Science Questions
Authors: Johannes Welbl, Nelson F. Liu, Matt Gardner
Link: https://arxiv.org/abs/1707.06209
Category: Evaluation / Question Answering Datasets

Key points

The paper introduces SciQ, a new crowdsourced dataset of multiple-choice science questions.
It demonstrates that, contrary to common assumptions, crowdsourced workers can write high-quality, exam-style science questions when given appropriate task design and quality control.
The dataset merges crowdsourced questions with existing standardized exam questions.
Each question includes a correct answer and three distractors, and is additionally annotated with:
An evidence paragraph supporting the correct answer, and
A difficulty rating predicted by an automatic question difficulty model.
These extra signals allow researchers to study distractor selection, difficulty modeling, and question generation—not just answer selection.

Motivation

Multiple-choice reading comprehension datasets drawn from standardized exams are limited in size because exam questions are expensive to author and often locked behind licensing restrictions. The authors explore whether crowdsourcing can scale up the creation of such questions while preserving the quality needed to challenge language understanding systems.

Dataset and Methodology

The authors design a crowdsourcing pipeline in which workers: 1. Read a science-related passage, 2. Write a question with a correct answer, and 3. Provide distractors, subject to quality filtering and validation.

Crowdsourced questions are combined with existing exam questions to form the full SciQ dataset, with each item enriched by evidence text and a predicted difficulty score. Readers should consult the original PDF for exact dataset statistics and baseline results.

Significance

SciQ has become a standard benchmark for evaluating machine comprehension and question answering models in the science domain. Its auxiliary annotations (evidence and difficulty) distinguish it from answer-only datasets and support broader research on question generation and data augmentation for QA.

Related resources

Evaluation of Retrieval-Augmented Generation: A Survey
A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts

> Original abstract: Crowdsourcing Multiple Choice Science Questions — see arXiv:1707.06209.