MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering

Forum topic · 小凯 · 2026-07-05

Summary

MedExpQA is a multilingual benchmark for evaluating large language models on medical question answering, introduced in a September 2024 paper published in Artificial Intelligence in Medicine (Elsevier). The benchmark is built from professional medical specialization examinations in four languages—Spanish, English, French, and Italian—yielding question sets that reflect real, high-stakes exam conditions rather than synthetic items. The authors evaluate a range of prominent LLMs, including GPT-3.5, GPT-4, Llama 2 variants, and medical-domain-tuned models such as Meditron, comparing performance across languages and exam sources. Experiments cover both zero-shot and few-shot settings and analyze answer accuracy and justification quality. A key finding is that model performance varies notably across languages, with English generally favored, revealing cross-lingual gaps that mono-lingual English benchmarks miss. The paper also explores ensemble and justification-based strategies to improve reliability. MedExpQA provides a reproducible evaluation resource for the medical NLP community and highlights the importance of multilingual, exam-derived evaluation when assessing LLM readiness for clinical and medical knowledge tasks. Source: sciencedirect.com/science/article/pii/S0933365724001805.

MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering (Sep 2024, Artificial Intelligence in Medicine)

> Source: ScienceDirect – Artificial Intelligence in Medicine

Key points

Motivation: Existing medical QA benchmarks are heavily English-centric. MedExpQA addresses this by evaluating LLMs in four languages: Spanish, English, French, and Italian.
Data construction: Questions are drawn from real medical specialization examinations (professional residency-style exams), producing authentic, high-difficulty test items rather than synthetic or crowdsourced questions.
Models evaluated: Prominent general-purpose LLMs (e.g., GPT-3.5, GPT-4, Llama 2) and medical-domain-tuned models (e.g., Meditron) are compared under zero-shot and few-shot prompting settings.
Findings: Performance is not uniform across languages—models tend to perform better in English, exposing cross-lingual gaps that English-only benchmarks fail to capture. Answer justification quality is also analyzed alongside accuracy.
Contributions: A reproducible multilingual benchmark, systematic comparison of general vs. medical-specialized LLMs, and analysis of strategies (e.g., ensembling / justification-based evaluation) for improving answer reliability.

Relevance

MedExpQA is relevant to researchers studying medical NLP, multilingual LLM evaluation, and the deployment readiness of LLMs for clinical knowledge tasks. It complements English-only suites (e.g., MedQA, MedMCQA) by testing whether medical competence transfers across languages.

Notes on this forum post

The original Chinese forum entry is largely a boilerplate template; its substantive content is limited to the title, source link, and a stub abstract. This translation/summary is based on the paper's title, venue, and link preserved in the post; quantitative results should be verified against the original PDF.