NovelQA: A Benchmark for Long-Range Novel Question Answering (arXiv, Mar 2024)
Overview
NovelQA: A Benchmark for Long-Range Novel Question Answering (arXiv, March 2024) addresses question answering over full-length novels — documents that far exceed the input capacity of most language models and standard QA benchmarks.
- Source: https://arxiv.org/abs/2403.12766v1
- Authors: Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, et al. (11 authors total)
- Type: Academic paper / benchmark
- Category: Evaluation of search engines / long-context QA
- A benchmark of question–answer pairs grounded in full-length novels, requiring long-range evidence integration rather than single-passage lookup.
- Manual verification of questions to ensure quality and answerability from the source text.
- An evaluation revealing that models can degrade significantly when short-context performance does not transfer to full-book length, exposing the gap between nominal context windows and effective usable context.
- A reproducible resource for studying long-context understanding, book-level RAG, and multi-hop reasoning.
- Long-context vs. retrieval: NovelQA provides evidence for when passing an entire document to a model fails, motivating hierarchical retrieval, chunking, and agentic retrieval strategies over long corpora.
- Evaluation practice: The benchmark illustrates a broader trend of moving from static passage-level metrics (nDCG, MRR) toward task-level, evidence-integration evaluation of retrieval and generation pipelines.
- Engineering takeaway: Claimed context lengths do not equal effective context; systems built on LLMs over book-length or enterprise-length documents should be evaluated with benchmarks like NovelQA before deployment.
- Evaluation of Retrieval-Augmented Generation: A Survey, May 2024, arXiv
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
- Original paper: NovelQA: A Benchmark for Long-Range Novel Question Answering. arXiv, March 2024. https://arxiv.org/abs/2403.12766v1
Background and Motivation
Conventional QA benchmarks operate on short passages or single documents, which masks a key limitation of large language models (LLMs): their ability to reason over genuinely long inputs. Long novels require integrating information scattered across hundreds of thousands of tokens — characters, plot threads, and causal links that appear far apart in the text. NovelQA was created to measure this long-range comprehension ability directly, in a setting relevant to open-domain information access, document retrieval, and retrieval-augmented generation (RAG) systems that must handle whole-book corpora.