QASPER: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Authors: Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner Source: https://arxiv.org/abs/2105.03011 (arXiv, May 2021)
Overview
QASPER is a question answering dataset targeting information-seeking questions about research papers. Unlike synthetic QA datasets generated from templates or crowd workers with no domain expertise, QASPER grounds its questions in the real information needs of NLP practitioners.
Dataset Construction
- Scale: 5,049 questions over 1,085 natural language processing papers drawn from the Semantic Scholar corpus of NLP papers (papers with S2ORC full text).
- Question writing: Annotators (NLP practitioners) were shown a paper's title and abstract and asked to write a question they genuinely wanted answered about the full paper.
- Answering: Different annotators, given access to the entire paper, provided answers. Each answer is one of:
- an extractive span copied from the paper,
- a yes/no decision,
- free-form text (when the answer cannot be expressed as a span or yes/no), or
- unanswerable (the paper does not contain the answer).
- QASPER established a realistic, practitioner-driven benchmark for scientific paper QA and long-document reading comprehension.
- It exposed a clear gap between abstract-level and full-text-level understanding, motivating subsequent work on long-context encoders, retrieval-augmented reading, and scientific information extraction.
- It remains a standard evaluation set for models targeting scholarly documents.
- Paper: https://arxiv.org/abs/2105.03011
- Related evaluation literature: Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437), ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
Baseline Models
The authors adapt Longformer-based reading comprehension models for the different answer types over long documents, including:
1. Extractive QA pipeline: span extraction over the full paper, handling yes/no and unanswerable cases as additional prediction options. 2. Filtering/pipeline baselines: using only title and abstract versus full text, showing that access to full-text content is critical for good performance; title/abstract-only models lag substantially behind.