Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey (arXiv 2503.22458)

Forum topic · 小凯 · 2026-07-05

Summary

This survey (arXiv:2503.22458, by Guan, Wang, Bian, Zhu, Lou, and Xiong, March 2025) systematically examines evaluation methods for LLM-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, the authors reviewed nearly 250 scholarly sources and developed two interrelated taxonomies: one defining what to evaluate (task completion, response quality, user experience, memory and context retention, planning and tool integration) and one explaining how to evaluate (annotation-based evaluation, automated metrics, hybrid strategies, and LLM self-judging methods). The survey covers both traditional language metrics such as BLEU and ROUGE and newer process-oriented metrics reflecting the dynamic, interactive nature of multi-turn dialogue. It also situates the work in the broader evolution from dense retrieval, late interaction, and generative IR to agentic search, discussing open problems including evaluation reliability, latency and cost, hallucination and safety, and multilingual/multimodal extension. Key insights for practitioners include the widening gap between offline metrics and online user satisfaction, and the need to cross-validate LLM-as-judge scores with human assessment.

Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey

arXiv: https://arxiv.org/abs/2503.22458
Authors: Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-guang Lou, Haoyi Xiong
Date: 2025-03-28
Type: Survey

Key Points

The survey systematically examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, the authors reviewed nearly 250 scholarly sources across venues, establishing a broad foundation for analysis.

Two Interrelated Taxonomies

1. What to evaluate — key components and their evaluation dimensions:

Task completion
Response quality
User experience
Memory and context retention
Planning and tool integration

2. How to evaluate — methodology categories:

Annotation-based (human) evaluations
Automated metrics (e.g., BLEU, ROUGE)
Hybrid strategies combining human and quantitative measures
Self-judging methods using LLMs (LLM-as-judge)

Context in the Broader Landscape

The survey sits at the intersection of agentic search and large-scale search/recommendation systems. The evolution of neural information retrieval can be traced through four main lines:

Dense retrieval: high recall, low latency; suited to first-stage retrieval
Late interaction (e.g., ColBERT): higher precision but larger indexes
Generative IR: directly "generates" documents via tokens or docids, simplifying the cascade
Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection

Common Benchmarks and Metrics Covered

Datasets: MS MARCO, BEIR, Natural Questions, domain corpora
Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost
Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs

Open Problems

Lack of unified benchmarks; limited reproducibility on private data
Bias in LLM-based evaluation; divergence between offline metrics and real user satisfaction
Safety, hallucination, latency, and cost constraints for agentic systems
Cross-lingual and multimodal generalization (English-centric data)

Takeaways for Practitioners

1. Architecture: cascade retrieval + rerank + generation remains mainstream, but agentic paradigms make retrieval strategy itself learnable 2. Data: high-quality instruction data and interaction logs matter; synthetic data risks knowledge leakage and distribution shift 3. Evaluation: cross-validate LLM-as-judge with human assessment; report statistical significance and compute cost 4. Product: latency, cost, explainability, and safety are hard constraints for deployment — do not optimize academic benchmarks alone

Original Abstract (English)

> This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines *what to evaluate* and another that explains *how to evaluate*. The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues.