NovelQA: A Benchmark for Long-Range Novel Question Answering (arXiv, Mar 2024)

Overview

This forum post indexes an academic paper from arXiv:

• Paper: NovelQA: A Benchmark for Long-Range Novel Question Answering
• Authors: Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, et al. (11 authors total)
• Link: https://arxiv.org/abs/2403.12766v1
• Section: Evaluation of Search Engines

Based on the title and metadata, the paper proposes a question answering benchmark grounded in full-length novels — documents substantially longer than those covered by most existing QA benchmarks. Such a benchmark is designed to test long-range comprehension, cross-chapter reasoning, and the effective context handling of large language models (LLMs) and retrieval-augmented systems on very long inputs.

> Note: The body of the original forum post is largely a generic reading-notes template with placeholder content. For concrete details — dataset construction, question types, evaluation metrics, and model results — consult the original paper PDF at the arXiv link above.

Why long-range novel QA matters

Within the broader context of LLM-era information retrieval and RAG (Retrieval-Augmented Generation) evaluation:

• Most QA benchmarks (e.g., Natural Questions, MS MARCO) operate on short passages or single web pages.
• Novels span hundreds of thousands of tokens, requiring models to integrate information across distant chapters, track characters and plot threads, and handle narrative-style rather than factual-style text.
• Long-context benchmarks probe whether models truly use their advertised context windows, and whether retrieval pipelines can route the right evidence to the generator.

Related resources referenced in the post

The entry cross-links several related evaluation works:

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
• ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv 2401.13131)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv 2105.03011)
• AI Search Has A Citation Problem (CJR, Mar 2025)

Takeaways for practitioners

1. Evaluation coverage: long-document QA is a distinct capability from passage-level QA; benchmarks like NovelQA help expose failures of long-context models and RAG pipelines on narrative text. 2. Verify against the source: quantitative claims in this post are template placeholders — cite numbers only from the original paper. 3. Positioning: this work fits into the chain of "retrieval → ranking → generation/agents → evaluation" literature indexed by the surrounding list.