MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
Paper: arXiv:2401.15391 Authors: Yixuan Tang, Yi Yang Published: 2024-01-27 Code: https://github.com/yixuantt/MultiHop-RAG/
Background
Retrieval-augmented generation (RAG) augments large language models (LLMs) with retrieved knowledge, showing promise in mitigating hallucinations and improving response quality. However, existing RAG systems are inadequate for multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Prior to this work, no RAG benchmarking dataset focused on multi-hop queries.
The MultiHop-RAG Dataset
The dataset consists of:
- A knowledge base built from an English news article corpus;
- A large collection of multi-hop queries;
- Ground-truth answers for each query;
- The associated supporting evidence.
- Multi-hop retrieval and reasoning remain a significant weakness of current RAG pipelines, even with frontier LLMs.
- MultiHop-RAG provides a reproducible benchmark covering both the retrieval stage (embedding comparison) and the generation stage (LLM reasoning).
- The dataset and RAG implementation are open-sourced, lowering the barrier for follow-up research.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv:2105.03011)
- Tang, Y., & Yang, Y. MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries. arXiv:2401.15391.
The paper details the full dataset construction procedure.
Experiments
Two benchmarking experiments demonstrate the dataset's utility:
1. Evidence retrieval: Comparing different embedding models for retrieving evidence for multi-hop queries. 2. Reasoning over evidence: Evaluating state-of-the-art LLMs (GPT-4, PaLM, Llama2-70B) on reasoning and answering multi-hop queries given the evidence.
Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries.
Original Abstract (English)
> Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. [...] We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice.