MTRAG: A Multi-Turn Conversational Benchmark for Evaluating RAG Systems
Source: arXiv:2501.03468 | Code: github.com/ibm/mt-rag-benchmark
Overview
Retrieval-augmented generation (RAG) has become a popular task for Large Language Models (LLMs), but most evaluations focus on single-turn, standalone questions. MTRAG addresses an important and often overlooked setting: multi-turn RAG conversations, where the system must answer a question in the context of a preceding conversation.
Key facts
- Type: End-to-end, human-generated multi-turn RAG benchmark covering the full RAG pipeline
- Scale: 110 conversations, averaging 7.7 turns each
- Coverage: Four domains, 842 total tasks
- Authors: Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, et al. (IBM)
- Published: 2025-01-07
- Later-turn questions that depend on conversation history
- Non-standalone questions requiring context resolution (e.g., pronouns, ellipsis)
- Unanswerable questions, testing whether systems refuse appropriately
- Multiple domains, testing generalization across topics
- A Survey on Multi-Turn Interaction Capabilities of Large Language Models (arXiv:2501.09959)
- Evaluating LLM-based Agents for Multi-Turn Conversations (arXiv:2503.22458)
- CHIQ: Contextual History Enhancement for Improving Query Rewriting (arXiv:2406.05013)
- Beyond Whole Dialogue Modeling: Contextual Disentanglement for Conversational IR (arXiv:2504.17427)
Real-world challenges captured by the benchmark
MTRAG deliberately includes properties that make evaluation realistic:
Automation paths
Beyond the human-generated data, the paper explores two automation directions:
1. Synthetic data for constructing multi-turn conversations at scale 2. LLM-as-a-Judge evaluation to reduce dependence on human raters
Findings
Human and automatic evaluations both show that even state-of-the-art LLM RAG systems struggle on MTRAG. The results demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains.
Original abstract
> Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.