MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

Overview

MTRAG (arXiv: 2501.03468, published 2025-01-07) is an end-to-end, human-generated benchmark for evaluating multi-turn Retrieval-Augmented Generation (RAG) systems, created by Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, and colleagues at IBM.

Background and Motivation

RAG has become a popular task for Large Language Models (LLMs), but most evaluation focuses on single-turn question answering. Multi-turn RAG—answering a question in the context of a preceding conversation—adds several challenges:

• Non-standalone questions that require conversational context to resolve
• Unanswerable questions the system should refuse rather than hallucinate
• Later turns that depend on earlier dialogue state
• Multiple domains in a single benchmark

Benchmark Design

MTRAG reflects real-world properties across diverse dimensions for evaluating the full RAG pipeline:

• 110 conversations, averaging 7.7 turns each
• Four domains covered
• 842 tasks in total
• Fully human-generated, ensuring natural conversation flow and realistic query formulation

The authors also explore automation paths to reduce the cost of benchmark construction and evaluation:

• Synthetic data generation
• LLM-as-a-Judge automatic evaluation

Key Findings

Human and automatic evaluations consistently show that even state-of-the-art LLM RAG systems struggle on MTRAG. The results demonstrate the need for stronger retrieval and generation systems that can handle:

1. Questions in later conversation turns 2. Unanswerable questions 3. Non-standalone, context-dependent questions 4. Multiple domains

Resources

• Paper: https://arxiv.org/abs/2501.03468
• Benchmark: https://github.com/ibm/mt-rag-benchmark

Original Abstract

> Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.

Why It Matters

For search and recommendation practitioners, MTRAG shifts evaluation beyond static metrics like nDCG toward conversation-level task success. It is a useful reference point alongside related work on multi-turn LLM interaction, query rewriting, and conversational recommendation systems.