BookQA: Stories of Challenges and Opportunities (October 2019, arXiv)

Authors: Stefanos Angelidis, Lea Frermann, Diego Marcheggiani, Roi Blanco, Lluís Màrquez Source: https://arxiv.org/abs/1910.00856 Type: Academic paper / dataset paper Category: Information retrieval, question answering, machine reading comprehension

One-Sentence Summary

BookQA is a large-scale question answering dataset centered on books and fictional narratives, highlighting the challenges and opportunities of answering real reader questions against long-form story content.

Research Background and Motivation

Most question answering (QA) research focuses on short passages, Wikipedia-style factoid questions, or well-structured knowledge sources. Books — especially works of fiction — present a fundamentally harder setting:

• Extreme document length: a full novel is far beyond the context capacity of typical reading comprehension models of the era;
• Natural, open-ended questions: real readers ask about plots, character motivations, and causal relationships, often requiring multi-hop reasoning across the entire narrative;
• Subjectivity and ambiguity: unlike factoid QA, questions about stories may admit multiple defensible answers;
• Answer extraction difficulty: relevant evidence is scattered and requires understanding of narrative structure.

BookQA was created to expose these challenges and to serve as a testbed for long-document QA.

Key Contributions

• A large-scale QA dataset of reader questions about books, collected from an online book community, each linked to the source book;
• An analysis of the characteristics of naturally occurring reader questions, in contrast to crowdsourced datasets;
• Baseline approaches demonstrating that existing retrieval and reading comprehension techniques fall short on this data;
• A discussion of the opportunities narrative QA opens for information retrieval and natural language understanding.

Method / Approach

The work follows a standard QA evaluation pipeline adapted to books:

1. Question collection — real questions posed by readers are harvested and paired with the books they concern; 2. Retrieval — evidence is retrieved from book content (e.g., abstracts or full narratives); 3. Answer prediction — reading comprehension or candidate selection models score potential answers; 4. Evaluation — model outputs are compared against reference answers to quantify the difficulty of the task.

Experiments and Evaluation

• Dataset: BookQA reader questions over book texts;
• Baselines: retrieval-based and reading comprehension baselines existing at the time;
• Findings: baselines perform substantially below human ability, confirming that book-length narrative QA is an open challenge.

For exact numerical results, consult the original PDF via the arXiv link above.

Conclusions and Insights

• Real reader questions are a valuable, low-cost source of supervision for narrative QA;
• Passage-level QA techniques do not transfer trivially to book-scale documents;
• Narrative comprehension requires handling long-range dependencies, character tracking, and causal reasoning — pushing beyond standard extractive QA.

Limitations and Future Work

The authors point to the scale mismatch between questions and books, evaluation subjectivity, and the need for better long-document models. Future directions include more efficient retrieval over full narratives, abstractive answering for subjective questions, and grounding models' answers in narrative evidence.

Cross-References in This Collection

• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
• Evaluation of Retrieval-Augmented Generation: A Survey
• ARES: An Automated Evaluation Framework for RAG

References

• Original paper: BookQA: Stories of Challenges and Opportunities — arXiv:1910.00856

Glossary

| Term | Meaning | |------|---------| | QA | Question Answering | | MRC | Machine Reading Comprehension | | IR | Information Retrieval | | Narrative QA | QA over stories, novels, and other long-form fiction |