NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Source: https://arxiv.org/abs/2403.12766
Metadata
| Field | Content | |-------|---------| | Title | NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens | | Authors | Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, et al. (11 authors in total) | | Link | https://arxiv.org/abs/2403.12766 | | Type | Academic paper | | Section | Evaluation of Search engines |
Background and Motivation
In large-scale search, recommendation, and personalization systems, information retrieval has long faced challenges in efficiency, scalability, and user intent understanding. Traditional pipeline-style approaches often treat retrieval, ranking, and generation as separate stages, making it hard to meet the LLM-era demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. NovelQA is proposed against this backdrop, aiming to systematically advance the theory and practice of question answering over very long documents (novels exceeding 200K tokens).
The core scenarios considered include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommender systems, and end-to-end architectures that couple external knowledge sources with generative models.
Core Contributions
- Provides a unified perspective for the problem domain, bringing scattered related work into a comparable framework.
- Decomposes method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) clearly for engineering practice.
- Offers reproducible benchmarks, datasets, or taxonomy tables, lowering the entry cost for follow-up researchers.
- Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, indicating paths from research prototypes to industrial systems.
- Lists open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation datasets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of retrieval steps, reranking depth, and training data scale to final quality.
- Evaluation of Retrieval-Augmented Generation: A Survey
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Method / System Architecture
The typical workflow follows four steps: problem formalization → model/system design → training or construction → inference pipeline.
1. Input and representation: encode queries, documents, and user context into dense or sparse representations, or structured prompts. 2. Core modules: retrievers, rerankers, planners, memory modules, tool interfaces, chained or parallelized per task. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), bootstrapped data synthesis. 4. Inference strategies: single-turn retrieval, iterative retrieval, parallel sub-queries, early stopping and budget control.
Experiments and Evaluation
Typical evaluation setups include:
Note: exact quantitative results should be verified against the original PDF.
Key Insights for Search / Rec / Personalization
1. Architecture: cascade retrieval + reranking + generation remains mainstream, but agentic paradigms are making "number and strategy of retrievals" itself a learnable object. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Products: latency, cost, explainability, and safety policies are hard constraints for industrial deployment—academic benchmarks alone are not enough.
Limitations and Future Work
Potential limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, unknown cross-lingual generalization due to English-centric data, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommender systems.
Engineering Checklist
| Item | Question | Recommendation | |------|----------|----------------| | Data | Does training/indexing contain PII? Version management? | Partitioned indexes, anonymization, rollback-safe embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stopping, cache hot queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU usage per query? | Route to smaller models, distillation, hybrid sparse+dense |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |