NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

Forum topic · 小凯 · 2026-07-05

Summary

NovelQA is a benchmark introduced in March 2024 (arXiv:2403.12766) for evaluating question answering over documents that exceed 200K tokens, targeting full-length novels as representative long-form contexts. The benchmark, developed by Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, and colleagues, addresses the gap between standard QA datasets built on short passages and the practical need for systems that reason over very long documents. By anchoring questions in complete novels, NovelQA tests multi-hop reasoning, cross-chapter information integration, and long-range understanding capabilities that passage-level benchmarks cannot measure. It serves as a tool for comparing large language models with long-context windows, retrieval-augmented pipelines, and agentic retrieval strategies, complementing related evaluation work such as retrieval-augmented generation surveys, ARES, and AgentBoard. This forum post provides metadata, a structured overview of the benchmark's motivation, methodology, and evaluation design, plus cross-references to related datasets and evaluation frameworks in the information retrieval and LLM evaluation space. Readers interested in long-context LLM evaluation, document-level QA, and agentic search should consult the original paper for quantitative results.

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

Overview

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens is an academic paper released in March 2024 on arXiv (https://arxiv.org/abs/2403.12766). The work was authored by Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, and 5 other collaborators.

Motivation

Most question answering benchmarks are built on short passages of a few hundred tokens, which fails to reflect the real-world demand for systems that can read, index, and reason over entire documents such as books, legal contracts, or technical reports. NovelQA addresses this gap by constructing a QA benchmark where each question is anchored in a full-length novel exceeding 200K tokens.

What the Benchmark Tests

Long-range understanding: questions require integrating information spread across distant chapters.
Multi-hop reasoning: answers depend on combining clues from multiple locations in the text.
Context window vs. retrieval: the benchmark enables comparison between long-context LLMs (reading the whole novel at once) and retrieval-augmented or agentic pipelines (iteratively retrieving relevant passages).

Positioning in the Field

This entry belongs to the Evaluation of Search Engines section of the forum's awesome list. It fits along the "retrieval → ranking → generation/agentic → evaluation" chain alongside related resources:

Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv:2105.03011)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13148)

Takeaways for Practitioners

1. Researchers: use NovelQA to stress-test long-context models against retrieval-based baselines; verify statistical significance and compute costs when citing results. 2. Engineers: benchmark pluggable components (retrievers, rerankers, planners) on document-scale QA before integration; pay attention to p99 latency budgets and index update strategies. 3. Product teams: user-perceivable quality on long documents (answer faithfulness, citation accuracy) matters more than passage-level nDCG improvements.

Caveats

This post summarizes the benchmark based on its abstract and public metadata. Quantitative results should be verified against the original PDF before citation. Note that novel-centric English benchmarks may not generalize across languages or document genres (e.g., enterprise or multimodal documents).

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

Summary

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

Overview

Motivation

What the Benchmark Tests

Positioning in the Field

Takeaways for Practitioners

Caveats

Tags