RAGAs: Automated Evaluation of Retrieval Augmented Generation (EACL 2024 Demo)
Source: https://aclanthology.org/2024.eacl-demo.16/
Key points
- RAGAs is a framework for reference-free, automated evaluation of Retrieval Augmented Generation (RAG) pipelines, introduced as a demo paper at EACL 2024.
- It uses large language models as evaluators to compute component-level metrics, avoiding expensive human annotation of golden answers for every pipeline iteration.
- Faithfulness: measures whether the generated answer is factually consistent with the retrieved contexts (hallucination detection).
- Answer relevancy: checks whether the generated answer actually addresses the input question.
- Context precision: evaluates whether relevant passages among the retrieved contexts are ranked highly.
- Context recall: evaluates whether the retrieved contexts cover the ground-truth answer (requires reference answers).
- Combining these metrics yields a holistic score covering both the retrieval and generation stages, helping developers localize failure modes.
- An open-source Python implementation accompanies the paper and integrates with common RAG tooling for practical iterative development.
- Classical pipelines separate retrieval, re-ranking, and generation; LLM-era systems additionally treat inference budget and action space (whether and how often to retrieve, which tools to call) as design variables.
- RAG and agentic search extend external knowledge access from one-shot retrieval to iterative, verifiable, plannable processes — so evaluation shifts from static metrics like nDCG toward task success rate, citation accuracy, and multi-hop reasoning chain completeness.
- LLM-as-judge approaches should be cross-validated with human evaluation, since offline metrics increasingly diverge from online user satisfaction.
- IR — Information Retrieval
- RAG — Retrieval-Augmented Generation
- LTR — Learning to Rank
- nDCG — Normalized Discounted Cumulative Gain
- Agentic Search — Search modeled as sequential decision-making with tool calls
- Gen-IR — Generative Information Retrieval
- A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models
- A Survey on Retrieval-Augmented Text Generation for Large Language Models
- Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
- RAFT: Adapting Language Model to Domain Specific RAG
- RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
Context in the RAG / Search landscape
The forum post situates RAGAs within the broader evolution of neural information retrieval and LLM systems:
Engineering checklist highlighted in the post
| Area | Concern | Suggestion | |------|---------|------------| | Data | PII in training/index data, embedding version control | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget, number of retrieval hops | Cascading with early stopping, caching hot queries, async re-ranking | | Quality | Does offline gain transfer to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias via open retrieval | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage | Small-model routing, distillation, hybrid sparse+dense retrieval |