LLMs Get Lost in Multi-Turn Conversation: A 39% Performance Drop Revealed

Forum topic · 小凯 · 2026-07-05

Summary

This post reviews the arXiv paper 'LLMs Get Lost in Multi-Turn Conversation' (Laban, Hayashi, Zhou, Neville; arXiv:2505.06120, May 2025). Through large-scale simulation of over 200,000 conversations across six generation tasks, the study shows that all tested top open- and closed-weight LLMs perform significantly worse in multi-turn settings than in single-turn, fully-specified instructions, with an average 39% performance drop. The degradation decomposes into a minor loss in aptitude and a major increase in unreliability: models make premature assumptions in early turns, over-rely on early partial solutions, and fail to recover from early mistakes. The article summarizes the research background, methodological framing, evaluation considerations, implications for RAG, search, recommendation and agentic systems, plus engineering checklists covering data, latency, quality, safety and cost. It is a useful reference for researchers and engineers building conversational LLM applications and evaluating multi-turn dialogue reliability.

LLMs Get Lost in Multi-Turn Conversation

This forum post summarizes the paper “LLMs Get Lost In Multi-Turn Conversation” (arXiv:2505.06120) by Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville (May 2025).

Original Abstract

> Large Language Models (LLMs) are conversational interfaces. As such, LLMs have the potential to assist their users not only when they can fully specify the task at hand, but also to help them define, explore, and refine what they need through multi-turn conversational exchange. Although analysis of LLM conversation logs has confirmed that underspecification occurs frequently in user instructions, LLM evaluation has predominantly focused on the single-turn, fully-specified instruction setting. In this work, we perform large-scale simulation experiments to compare LLM performance in single- and multi-turn settings. Our experiments confirm that all the top open- and closed-weight LLMs we test exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks. Analysis of 200,000+ simulated conversations decomposes the performance degradation into two components: a minor loss in aptitude and a significant increase in unreliability. We find that LLMs often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely. In simpler terms, we discover that *when LLMs take a wrong turn in a conversation, they get lost and do not recover*.

Key Findings

Multi-turn penalty: All tested frontier open- and closed-weight LLMs perform significantly worse in multi-turn conversations than in single-turn settings — an average 39% performance drop across six generation tasks.
Aptitude vs. reliability: Analysis of 200,000+ simulated conversations shows degradation consists of a *minor* loss of aptitude but a *significant* increase in unreliability.
Failure mechanism: LLMs often make assumptions in early turns and prematurely attempt final solutions, then over-rely on them. Once an early turn goes wrong, the model rarely recovers.
Evaluation gap: Prior LLM evaluation has focused almost exclusively on single-turn, fully-specified instructions, while real user instructions are frequently underspecified.

Context for RAG / Search / Recommendation Systems

The post situates the paper within the broader RAG and large-scale search/recommendation landscape, where key themes include:

1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, while agentic paradigms make retrieval count and strategy themselves learnable. 2. Data: High-quality instruction data and session logs matter; synthetic data requires protection against leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Productization: Latency, cost, explainability, and safety are hard constraints for industrial deployment.

Engineering Checklist (from the post)

| Aspect | Question | Suggestion | |--------|----------|------------| | Data | Does training/index data contain PII? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascades + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source allowlists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Glossary

IR: Information Retrieval
RAG: Retrieval-Augmented Generation
LTR: Learning to Rank
nDCG: Normalized Discounted Cumulative Gain
Agentic Search: Modeling search as sequential decision-making and tool use
Gen-IR: Generative Information Retrieval