Ragas: Automated Evaluation of Retrieval Augmented Generation

Overview

Ragas (Retrieval Augmented Generation Assessment) is a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines.

| Field | Detail | |-------|--------| | Authors | Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert | | Published | 2023-09-26 | | Source | https://arxiv.org/abs/2309.15217 | | Type | Academic paper |

Original Abstract

> We introduce Ragas (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAG systems are composed of a retrieval and an LLM based generation module, and provide LLMs with knowledge from a reference textual database, which enables them to act as a natural language layer between a user and textual databases, reducing the risk of hallucinations. Evaluating RAG architectures is, however, challenging because there are several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages in a faithful way, or the quality of the generation itself. With Ragas, we put forward a suite of metrics which can be used to evaluate these different dimensions *without having to rely on ground truth human annotations*. We posit that such a framework can crucially contribute to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.

Why It Matters

• RAG evaluation spans several dimensions: retrieval relevance, faithfulness of the LLM to the retrieved context, and overall generation quality.
• Ragas provides metrics for all these dimensions without ground-truth human annotations, enabling rapid, automated evaluation loops.
• Faster evaluation cycles are critical given how quickly LLM-based systems are being adopted.

Related Work Context

Neural information retrieval has evolved from BM25 to dense dual-tower retrieval, cross-encoder reranking, late interaction, generative retrieval, and LLM agents. RAG and agentic search extend external knowledge access from a single retrieval step into an iterative, verifiable, plannable process — shifting evaluation from static nDCG toward task success rate, citation accuracy, and multi-hop reasoning chain completeness.

Engineering Checklist

| Concern | Question | Recommendation | |---------|----------|----------------| | Data | PII in training/index data? Version control? | Partitioned indexes, anonymization, rollback-safe embedding versions | | Latency | p99 budget? Retrieval steps? | Cascading + early stopping, query caching, async reranking | | Quality | Do offline gains transfer to online satisfaction? | Interleaving experiments, human audit samples, citation checks | | Security | Poisoning/bias via open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Related Entries

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
• ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv 2105.03011)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv 2401.13178)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |