The NarrativeQA Reading Comprehension Challenge (DeepMind, arXiv 1712.07040)

Overview

This post is an index entry for the DeepMind paper "The NarrativeQA Reading Comprehension Challenge", published on arXiv in December 2017.

• Paper: https://arxiv.org/abs/1712.07040
• Authors: Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and colleagues (7 authors total)
• Category: Evaluation of search engines / question answering benchmarks

Key points

• NarrativeQA is a reading comprehension benchmark over whole narratives: full stories from books and movie screenplays, paired with human-written question–answer dialogues.
• Annotators composed questions and answers while viewing only story summaries, so answers cannot be extracted verbatim from a single passage; a system must read and understand the entire narrative to answer well.
• The benchmark is therefore much harder than span-extraction datasets (e.g., SQuAD, CNN/Daily Mail), where answers appear as contiguous text snippets.
• The paper reports baseline results for several existing models and shows substantial headroom between machine and human performance, motivating research on long-document comprehension.
• The dataset and evaluation code were released publicly to support reproducible comparison of models.

Context in IR / QA research

The post places NarrativeQA in the broader lineage of neural information retrieval and QA evaluation, alongside later datasets and frameworks such as:

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (arXiv:2105.03011)
• ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)

Within a retrieval → ranking → generation → evaluation pipeline, NarrativeQA represents the long-document, abstractive end of evaluation: it tests whether a system can integrate evidence distributed across an entire narrative rather than rerank snippets.

Notes and caveats

• The post's descriptive sections are largely boilerplate metadata; the substantive content is the paper reference itself.
• Quantitative results (dataset sizes, metric scores such as ROUGE-L / METEOR / BLEU) should be taken from the original PDF rather than this summary.

Reference

• Original paper: The NarrativeQA Reading Comprehension Challenge. arXiv:1712.07040, December 2017. https://arxiv.org/abs/1712.07040