BookQA: Stories of Challenges and Opportunities (Oct 2019, arXiv)
Metadata
| Field | Value | |-------|-------| | Title | BookQA: Stories of Challenges and Opportunities | | Authors / Affiliations | Stefanos Angelidis, Lea Frermann, Diego Marcheggiani, Roi Blanco, Lluis Marquez | | Published | October 2019 | | Source | https://arxiv.org/abs/1910.00856 | | Type | Academic paper | | Section | Evaluation of Search engines |
One-line summary
This work introduces BookQA, a question answering dataset built over long narrative stories, and examines the challenges and opportunities of machine comprehension at book scale.
Background and Motivation
Most reading comprehension benchmarks operate over short passages, which limits their ability to test genuine narrative understanding. Long-form stories require tracking characters, events, temporal ordering, and plot structure across thousands of sentences, well beyond the context typical of paragraph-level QA. BookQA was proposed to address this gap by grounding questions and answers in full-length narrative books, creating a testbed for story-level comprehension and for evaluating retrieval and reading systems on long documents.
Core Contributions
- A question answering dataset anchored in long narrative texts (children's stories), requiring document-level understanding rather than local span extraction.
- An analysis of the challenges specific to book-scale QA: long-range dependencies, discourse and character modeling, and evaluation methodology.
- An assessment of the gap between human performance and automated baselines on narrative QA, highlighting opportunities for future work.
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts
- Evaluation of Retrieval-Augmented Generation: A Survey
- ARES: An Automated Evaluation Framework for RAG
- AI Search Has A Citation Problem (CJR, Mar 2025)
- Original paper: BookQA: Stories of Challenges and Opportunities. arXiv:1910.00856.
Method / Setup
The work follows the standard dataset-paper pipeline: source narrative documents are collected and processed; questions and answers are constructed to require understanding of the full story; baseline systems are evaluated against the dataset, with results compared to human performance to quantify the difficulty of the task.
Evaluation
As is typical for QA benchmark papers, evaluation considers automatic answer-matching metrics and human performance as an upper bound. Exact numbers should be verified against the original paper tables; this post is based on the abstract and public metadata.
Takeaways for Search / Rec / Personalization
1. Long-context evaluation: paragraph-level benchmarks understate the difficulty of true narrative comprehension; book-scale datasets like BookQA complement them. 2. Relevance to RAG: narrative QA anticipates current challenges in retrieval-augmented generation, where documents exceed context windows and multi-hop, document-level reasoning is required. 3. Evaluation design: answer validation and question quality matter as much as model architecture when building long-document QA benchmarks.