The NarrativeQA Reading Comprehension Challenge (DeepMind, arXiv 1712.07040)

Forum topic · 小凯 · 2026-07-05

Summary

This forum post indexes the NarrativeQA Reading Comprehension Challenge, a dataset and benchmark introduced by Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis and colleagues (DeepMind, December 2017, arXiv:1712.07040). NarrativeQA is designed to test reading comprehension over whole narratives: it pairs real books and movie scripts with questions written by humans, and — crucially — the reference answers are written by annotators rather than extracted as spans from the text. This forces models to truly understand long-form narratives instead of relying on shallow span-matching heuristics that worked on earlier datasets. The post places the paper within an information-retrieval and search-evaluation collection, alongside related resources on retrieval-augmented generation evaluation, citation attribution in AI search, and LLM agent benchmarks. It also offers an engineering-oriented checklist covering data privacy, latency budgets, quality verification, safety against retrieval poisoning, and cost control, plus a glossary of IR/RAG terminology. Readers are directed to the original arXiv page for the authoritative abstract and quantitative results.

The NarrativeQA Reading Comprehension Challenge (DeepMind, Dec 2017)

Overview

This entry catalogs an academic paper from an information-retrieval evaluation collection:

Title: The NarrativeQA Reading Comprehension Challenge
Authors: Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, et al. (7 authors)
Link: https://arxiv.org/abs/1712.07040
Type: Academic paper / benchmark dataset (DeepMind, December 2017)
Category: Evaluation of Search Engines

About the Paper

NarrativeQA is a reading comprehension benchmark built on whole narratives — full-length books and movie scripts — paired with human-written questions. Unlike earlier datasets where answers are extractive spans copied from the source text, NarrativeQA requires abstractive answers: human annotators wrote the reference answers independently, so models cannot succeed by exploiting superficial lexical overlap and must genuinely comprehend long-form stories.

The authors provide baselines showing that state-of-the-art (at the time) reading comprehension systems performed far below human performance on this task, establishing NarrativeQA as a challenging testbed that drove subsequent progress in long-document understanding, abstractive QA, and narrative reasoning.

Position in the Collection

The post groups this entry with related resources on evaluation of modern search and QA systems, including:

Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles (arXiv:2105.03011)
ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
AI Search Has A Citation Problem (CJR, Mar 2025)

Engineering Notes (from the post's appendix)

The poster includes a general engineering checklist relevant when reproducing or deploying QA/IR systems:

| Area | Concern | Recommendation | |------|---------|----------------| | Data | PII in training/index, versioning | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget, retrieval depth | Cascaded retrieval + early stopping, query caching, async reranking | | Quality | Offline gains vs. online satisfaction | Interleaving experiments, human audit, citation verification | | Safety | Poisoning/bias in open retrieval | Source allowlists, adversarial detection, output filtering | | Cost | Token and GPU cost per query | Model routing, distillation, hybrid sparse+dense retrieval |

Terms

IR: Information Retrieval
RAG: Retrieval-Augmented Generation
Agentic Search: Modeling search as sequential decision-making with tool calls
Gen-IR: Generative Information Retrieval

> Note: Quantitative results should be verified against the original PDF at https://arxiv.org/abs/1712.07040.