R2MED: A Benchmark for Reasoning-Driven Medical Retrieval (arXiv, May 2025)
- Paper: https://arxiv.org/abs/2505.14558
- Authors: Xiangxu Zhang, Lei Li, Xiao Zhou, Zheng Liu
- Category: Evaluation of Search Engines
- Type: Academic paper
- From BM25 to LLM agents: Neural IR has evolved from sparse retrieval through dense bi-encoders, cross-encoder rerankers, and late interaction, to generative retrieval and agentic search. Each generation trades off efficiency, effectiveness, and maintainability.
- Reasoning as part of retrieval: RAG and agentic search extend external knowledge access from one-shot retrieval to an iterative, verifiable, plannable process. Evaluation accordingly shifts from static nDCG to process metrics such as task success rate, citation accuracy, and multi-hop reasoning chain completeness.
- Domain stress-testing: Medical retrieval is a hard testbed because queries require domain expertise, precise terminology, and careful evidence synthesis — exactly where hallucination and retrieval failure carry the highest risk.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, task success rate, and increasingly process-oriented measures;
- Baselines: BM25, dense retrievers, cross-encoder rerankers, retrieval-free LLMs;
- Ablations: contribution of retrieval depth, reranking, and training data scale.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv 2105.03011)
What the paper is about
R2MED is a benchmark for reasoning-driven medical retrieval, proposed in May 2025 on arXiv. The work addresses a long-standing challenge in large-scale search and recommendation systems: traditional pipelines split retrieval, ranking, and generation into isolated stages, which struggles to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and up-to-date domain knowledge. R2MED focuses specifically on the medical domain, where retrieving relevant evidence requires reasoning over specialized knowledge rather than simple lexical or semantic matching.
Why it matters for search and RAG
The benchmark lands at the intersection of several active research threads:
Typical evaluation design in this space
Benchmarks of this kind are commonly assessed with:
Concrete numbers for R2MED should be taken from the original PDF rather than this digest.
Takeaways for practitioners
1. Architecture: Cascaded retrieval + rerank + generation remains mainstream, but agentic paradigms make "whether and how many times to retrieve" itself a learnable decision. 2. Data: High-quality instruction data and synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human assessment. 4. Deployment: Latency, cost, explainability, and safety are hard constraints — academic benchmarks alone do not guarantee product value.