Natural Questions: A Benchmark for Question Answering Research (TACL 2019)
Overview
This post indexes the paper "Natural Questions: A Benchmark for Question Answering Research", published in *Transactions of the Association for Computational Linguistics (TACL)*, 2019.
- Source: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00276/43518/Natural-Questions-A-Benchmark-for-Question
- Resource type: Academic paper
- Category: Evaluation of Search Engines
- a long answer (the passage that contains the answer), and
- a short answer (the exact span, where one exists).
- Classic pipelines (BM25 recall → cross-encoder reranking → answer generation) were widely benchmarked on NQ.
- With LLM-era retrieval-augmented generation (RAG) and agentic search, evaluation is shifting from static ranking metrics (nDCG, MRR, Recall@k) toward task success rates, citation accuracy, and multi-hop reasoning quality — but datasets like NQ remain foundational reference points.
- Open problems the surrounding literature highlights include evaluation trustworthiness, latency/cost trade-offs, hallucination and safety, and cross-lingual/multimodal extension.
- Precise dataset statistics, baselines (e.g., end-to-end vs. reading-only models), and metric values should be taken from the original PDF rather than this summary — the source post is largely a templated index entry.
- The page includes an engineering checklist relevant to anyone building search/QA systems: PII handling in indexes, p99 latency budgets, offline-vs-online metric alignment, retrieval poisoning defenses, and per-query token/GPU cost control.
- AI Search Has A Citation Problem, Mar 2025, CJR
- Evaluation of Retrieval-Augmented Generation: A Survey, May 2024, arXiv:2405.07437
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Data (arXiv:2105.03011)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
Why It Matters
The Natural Questions (NQ) dataset addresses a long-standing gap in question answering research: most prior QA benchmarks used artificial or crowdsourced questions, which do not reflect how real users actually search. NQ instead samples real queries from Google Search and pairs each with a Wikipedia page, with human annotators marking:
This makes NQ a benchmark for end-to-end QA — systems must both retrieve the relevant document/passage and extract the answer — rather than answer extraction from an already-given context.
Relevance to Modern IR and RAG
The post situates NQ within the broader evolution of search and retrieval systems: