Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey (arXiv 2503.22458)

Forum topic · 小凯 · 2026-07-05

Summary

This survey examines evaluation methods for LLM-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, the authors systematically reviewed nearly 250 scholarly sources across venues, establishing a structured foundation for analysis. The paper contributes two interrelated taxonomies: one defining what to evaluate — covering task completion, response quality, user experience, memory and context retention, and planning/tool integration — and another defining how to evaluate, categorizing methodologies into annotation-based evaluation, automated metrics, hybrid human-plus-quantitative strategies, and LLM self-judging methods. The survey situates modern agent evaluation within the broader evolution of neural information retrieval, from BM25 and dense retrieval (DPR, ColBERT) to generative IR, RAG, and agentic search, and traces the field's timeline from BERT-era reranking through reinforcement-learning-trained search agents, Deep Research, and GraphRAG. It highlights open challenges including evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal generalization, and the gap between offline metrics and real user satisfaction. The post also provides engineering checklists for deployment (data privacy, p99 latency budgets, citation verification, retrieval poisoning defenses) and recommendations for researchers, engineers, and product managers.

Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey

Paper: arXiv:2503.22458 Authors: Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-guang Lou, Haoyi Xiong Published: 2025-03-28 Type: Survey

Abstract (original)

> This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines *what to evaluate* and another that explains *how to evaluate*. The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues.

Key Points

Scope: Systematic review of ~250 sources on evaluating LLM-based agents in multi-turn conversations, using a PRISMA-inspired methodology.
Two taxonomies:
*What to evaluate:* task completion, response quality, user experience, memory/context retention, planning and tool integration.
*How to evaluate:* annotation-based evaluation, automated metrics (BLEU, ROUGE, etc.), hybrid human+quantitative strategies, and LLM-as-judge / self-judging methods.

Context: The Broader Search/Retrieval Landscape

The post situates the survey within agentic search and large-scale retrieval systems. Classic pipelines separate retrieval, ranking, and generation; the LLM era introduces new variables — inference budget and action space (whether to retrieve, how many times, which tools to call).

Four main technical lines are compared:

Dense retrieval (bi-encoders): high recall, low latency; suitable for first-stage retrieval; sensitive to domain shift and long-tail queries.
Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
Generative IR (DSI, docid generation): simplifies cascades but faces index-update challenges.
Agentic search: models search as sequential decision-making with multi-hop reasoning and self-reflection.

Timeline of the Field

2019–2021: BERT reranking and DPR establish neural retrieval foundations.
2022–2023: RAG and FreshLLM drive retrieval-generation fusion.
2024: Conversational/agentic search and Gen-RecSys surge.
2025–2026: RL-trained search agents, Deep Research, and GraphRAG become growth frontiers.

Taxonomy Table

| Dimension | Sub-categories | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoder, cross-encoder, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization goals | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B | Comparable | Diverges from real satisfaction |

Evaluation Datasets and Metrics

Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
Baselines: BM25, dense retrieval, cross-encoder reranking, no-retrieval LLM, commercial search APIs.

Note: exact quantitative results should be verified against the original PDF.

Insights for Search / Recommendation / Personalization

1. Architecture: Cascade retrieval + reranking + generation remains dominant, but agentic paradigms make retrieval *strategy itself* learnable. 2. Data: High-quality instruction data and click/session logs are both critical; synthetic data requires guarding against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Product: Latency, cost, explainability, and safety are hard constraints for industrial deployment.

Open Problems

Lack of unified benchmarks; irreproducible private data.
LLM evaluation bias (LLM-as-judge limitations).
Safety and cost constraints of agentic systems on the open web.
Future directions: finer-grained process supervision, retrieval-reasoning co-training, enterprise metadata governance, multimodal and cross-lingual consistency, causal and fairness constraints for recommendation.

Engineering Checklist

| Check | Question | Suggestion | |---|---|---| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU usage per query? | Small-model routing, distillation, hybrid sparse+dense |

Glossary

IR — Information Retrieval
RAG — Retrieval-Augmented Generation
LTR — Learning to Rank
nDCG — Normalized Discounted Cumulative Gain
Agentic Search — Modeling search as sequential decisions and tool calls
Gen-IR — Generative Information Retrieval