RAGAs: Automated Evaluation of Retrieval Augmented Generation (EACL Demo 2024)
Overview
This entry catalogs RAGAs: Automated Evaluation of Retrieval Augmented Generation, a demo paper presented at EACL 2024.
- Source: https://aclanthology.org/2024.eacl-demo.16/
- Type: Academic paper (demo)
- Category: RAG
- A unified, reference-free evaluation framework for RAG pipelines.
- Component-wise metrics covering faithfulness, answer relevance, and context relevance, which can be combined into an overall score.
- Practical tooling that lowers the barrier for teams to benchmark and iterate on RAG systems.
- A articulation of open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora.
- Ranking metrics: nDCG@10, MRR, Recall@k, Hit@k.
- Baselines: BM25, dense retrieval, cross-encoder rerankers, retrieval-free LLMs, commercial search APIs.
- A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models
- A Survey on Retrieval-Augmented Text Generation for Large Language Models
- Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
- RAFT: Adapting Language Model to Domain Specific RAG
- RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
- Algolia's Knowledge Graphs and Ontologies — Adding Knowledge to Keyword Search
- Original paper: RAGAs: Automated Evaluation of Retrieval Augmented Generation, EACL Demo 2024. https://aclanthology.org/2024.eacl-demo.16/
RAGAs proposes a framework for reference-free, automated evaluation of Retrieval Augmented Generation pipelines, scoring both the retrieval and generation components without requiring ground-truth answers for every query.
Context and Motivation
In large-scale search, recommendation, and personalization systems, RAG has long faced challenges around efficiency, scalability, and intent understanding. Traditional pipelines treat retrieval, ranking, and generation as separate stages, which is increasingly at odds with LLM-era demands for natural language interaction, multi-hop reasoning, and real-time knowledge.
A key difficulty in RAG development is evaluation: offline metrics like nDCG measure ranking quality but not whether the final answer is faithful to the retrieved context or actually useful. RAGAs targets this gap by providing automated, LLM-assisted metrics that decompose RAG quality into interpretable dimensions such as faithfulness, answer relevance, and context relevance.
Core Contributions
Evaluation Landscape
Typical RAG evaluation setups in this space involve:
The post notes that RAG evaluation is shifting from static ranking metrics toward task success rate, citation accuracy, and multi-hop reasoning chain completeness — and that LLM-as-judge approaches should be cross-validated with human evaluation.
Key Insights for Search / Rec / Personalization
1. Architecture: cascaded retrieve → rerank → generate remains mainstream, but agentic paradigms make "whether and how often to retrieve" itself a learnable decision. 2. Data: high-quality instruction data and click/session logs are critical; synthetic data requires safeguards against leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; process-level metrics matter. 4. Production: latency, cost, explainability, and safety are hard constraints that cannot be optimized away in favor of academic benchmarks.
Engineering Checklist
| Item | Question | Recommendation | |------|----------|----------------| | Data | PII in training/index? Version control? | Partitioned indexes, sanitization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades + early stopping, query caching, async reranking | | Quality | Do offline gains translate online? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning / bias via open retrieval? | Source allowlists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Model routing, distillation, hybrid sparse+dense retrieval |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |