Evaluating Retrieval Quality in Retrieval-Augmented Generation (arXiv 2404.13781)
- Authors: Alireza Salemi, Hamed Zamani
- Published: April 2024
- Source: https://arxiv.org/abs/2404.13781
- Category: Evaluation of Search Engines / RAG
- Motivation: RAG pipelines couple retrieval with generation, but retrieval is typically evaluated with traditional information retrieval (IR) metrics that were designed for ranked result lists, not for downstream LLM consumption.
- Problem: There is a potential gap between retrieval effectiveness and generation utility — a document ranked lower by standard metrics may still be highly useful to the generator, and vice versa.
- Focus areas: the paper examines the correlation between retrieval quality signals and RAG output quality across datasets, and discusses evaluation methodologies suited to end-to-end RAG assessment.
- Evaluation of RAG has become an active subfield: related resources include *Evaluation of Retrieval-Augmented Generation: A Survey* (arXiv 2405.07437), *ARES: An Automated Evaluation Framework for RAG* (arXiv 2311.09476), and anchored QA datasets such as those in the QASPER line of work.
- Broader trends in search and recommendation push evaluation beyond static ranking metrics toward task success rate, citation accuracy, and multi-hop reasoning quality.
- Researchers: when comparing retrievers for RAG, consider end-to-end generation quality in addition to IR metrics, and report statistical significance and compute cost.
- Engineers: offline retrieval improvements may not translate into better answers; validate with interleaving experiments, human audits, and citation checks.
- Product teams: user-perceivable quality (latency, answer trustworthiness, multi-turn consistency) matters more than offline nDCG alone.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (arXiv 2105.03011)
- AI Search Has A Citation Problem (CJR, March 2025)
Overview
This paper studies how retrieval quality should be measured in retrieval-augmented generation (RAG) systems, where a retriever supplies evidence to a large language model (LLM) that generates the final answer. The central observation is that conventional retrieval metrics (e.g., nDCG, MRR) may not align with end-to-end generation quality, since what matters is whether retrieved documents help the generator, not whether they match human relevance labels.
Key points
Context within the field
Practical implications
Limitations and caveats
Quantitative results, exact datasets, and metrics are not reproduced in this summary; readers should verify all figures against the original paper PDF.