TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
Overview
TyDi QA (arXiv:2003.05002) is a 2020 benchmark for question answering across typologically diverse languages. It is listed in this collection under "Evaluation of Search engines".
Key points
- Authors: Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, et al. (7 authors total).
- Source: https://arxiv.org/abs/2003.05002
- Data collection: Questions are real, information-seeking queries written by native speakers; they were produced directly in the target languages without seeing English translations, reducing translationese bias common in multilingual QA datasets.
- Scale and coverage: More than 200k question-answer pairs spanning 11 typologically diverse languages.
- Tasks: Primary answer prediction, passage selection, and minimal-answer span extraction.
- Relevance: Useful as an evaluation benchmark for search engines, multilingual retrieval, and QA systems, including the retrieval → ranking → generation pipeline emphasized in LLM-era architectures.
- AI Search Has A Citation Problem (Mar 2025, CJR)
- Evaluation of Retrieval-Augmented Generation: A Survey (May 2024)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts
- ARES: An Automated Evaluation Framework for RAG
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Systems
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Related entries in this collection
Caveats of this post
The extended analysis sections in the original forum post (engineering checklists, terminology glossary, action recommendations) are template-generated and not specific to TyDi QA. Readers should consult the original PDF for exact experimental numbers and task definitions before citing quantitative results.