MedQA: A Large-scale Open Domain Question Answering Dataset from Medical Exams (Jin et al., 2020)

Forum topic · 小凯 · 2026-07-05

Summary

MedQA, introduced by Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits in the paper 'What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams' (arXiv:2009.13081, 2021), is a benchmark of multiple-choice medical questions collected from professional medical licensing examinations. The dataset targets open-domain question answering in the medical domain at a professional level, with questions sourced from exams in China and the United States, each accompanied by multiple candidate answers. MedQA was designed to enable standardized evaluation of models that must interpret patient presentations, clinical descriptions, and medical knowledge to select the correct diagnosis or answer. It quickly became one of the standard benchmarks—alongside MedMCQA and PubMedQA—for measuring biomedical question answering and clinical reasoning in large language models, including GPT-series and medical fine-tuned models. This forum post indexes the paper with metadata, related entries on retrieval-augmented generation evaluation, datasets, and agent benchmarks, and situates MedQA within a broader reading path covering retrieval, ranking, generation, and evaluation. Readers citing quantitative results are advised to consult the original PDF.

MedQA: A Large-scale Open Domain Question Answering Dataset from Medical Exams

Paper: Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, Peter Szolovits. *What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams.* arXiv, 2020. Link: https://arxiv.org/abs/2009.13081 Category: Evaluation of Search engines / medical QA benchmarks

Key points

MedQA is a large-scale open domain question answering dataset built from professional medical licensing exam questions, at a professional (physician) level of difficulty.
Questions are drawn from exams in China and the United States, formulated as multiple-choice questions with patient presentations and candidate diagnoses/answers.
The dataset provides a standardized benchmark for evaluating models on clinical reasoning and medical knowledge, complementing later benchmarks such as MedMCQA and PubMedQA.
MedQA is widely used to evaluate large language models and retrieval-augmented systems in the biomedical domain; it is a standard item in LLM medical capability evaluations.

Positioning within the IR / LLM landscape

This entry belongs to a curated list covering the retrieval → ranking → generation → evaluation pipeline. MedQA serves as a domain-specific evaluation resource, and pairs naturally with related entries in the same collection:

Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv:2105.03011, i.e., QASPER)
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation (arXiv:2311.09476)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)

Practical notes for readers

Researchers: use MedQA alongside other medical QA benchmarks; check whether reported scores include statistical significance and compute cost.
Engineers: when integrating retrieval into medical QA, evaluate failure modes (empty retrieval, wrong option selection) rather than accuracy alone.
Citation hygiene: quantitative numbers in this index entry are based on public metadata; verify exact dataset sizes and results against the original PDF before citing.

Reference

Original paper: Jin, Di, et al. "What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams." https://arxiv.org/abs/2009.13081