QASPER: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Overview
QASPER (Question Answering on Scientific Papers) was presented in the paper *"A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers"* (May 2021, arXiv:2105.03011), authored by Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner (Allen Institute for AI; University of Washington; Yale University).
Key Points
- Motivation: Most QA benchmarks operate over short passages, yet real information needs—especially in science—require reasoning over entire research papers. QASPER targets questions written by NLP practitioners who actually read the papers.
- Dataset composition: 5,049 questions, each tied to one of 1,585 papers drawn from the NLP domain (Semantic Scholar corpus).
- Answer types:
- *Extractive*: a span copied from the paper
- *Abstractive*: a free-form answer synthesized from the paper's content
- *Yes/no*: boolean questions
- *Unanswerable*: questions for which the paper contains no answer (roughly a quarter of the data)
- Evidence: Answers are grounded in specific evidence-containing sections of the paper, supporting evidence identification as an auxiliary task.
- Baseline models perform substantially worse than humans on both answer extraction and generation, indicating that scientific-document QA is far from solved.
- Performance degrades when models must reason across the full paper rather than oracle evidence.
- QASPER has since become a standard benchmark for scientific literature understanding, retrieval-augmented generation (RAG), and long-context language models.
- Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021). *A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers*. arXiv:2105.03011. https://arxiv.org/abs/2105.03011
Methods and Baselines
The authors adapt existing extractive and abstractive QA architectures to operate on full papers, using hierarchical/long-input encoders to handle document length. The paper formalizes tasks for answer extraction, answer generation, yes/no decision, and unanswerability detection.
Results and Takeaways
Relevance
QASPER sits at the intersection of information retrieval, scientific NLP, and question answering. It complements benchmarks like Natural Questions and MS MARCO by focusing on full-length scholarly documents, making it a natural evaluation target for systems that retrieve, read, and reason over research papers.