ICLR 2026 Best Paper: LLMs Get Lost in Multi-Turn Conversation — A Deep Dive
> Core finding up front: All mainstream LLMs (from Llama 3.1-8B to GPT-4.1 and Gemini 2.5 Pro) drop an average of 39% in performance in multi-turn conversations. The problem is not that models get "dumber" — it is a collapse in reliability. Once a model makes a wrong assumption in early turns, it sinks deeper and deeper like slipping into mud, and it cannot recover on its own.
1. Paper Overview
| Attribute | Content | |---|---| | Title | LLMs Get Lost In Multi-Turn Conversation | | Authors | Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville | | Affiliations | Microsoft Research, Salesforce Research | | Venue | ICLR 2026 (Best Paper Award) | | arXiv | 2505.06120 | | Core finding | 15 LLMs, 200K+ conversations, 6 generative tasks; average multi-turn performance drop of 39% |
2. Core Finding: Dissecting the 39% Drop
2.1 Methodology: Shard Simulation
The core method splits a fully-specified single-turn instruction into sharded multi-turn instructions:
- Fully-specified: all conditions and requirements given at once
- Sharded: the same information is split into "shards," revealing at most one per turn
- Single-turn: "Jay makes 20 snowballs per hour, 2 melt every 15 minutes; how long to accumulate 60?"
- Multi-turn: Turn 1: "Jay is making snowballs for a snowball fight" → Turn 2: "He can make 20 per hour" → Turn 3: "But 2 melt every 15 minutes" → Turn 4: "He needs 60 — how long?"
- Single-turn: stronger models show a small gap
- Multi-turn: every model shows an enormous gap — same model, same task, different conversation paths, wildly different results
- CONCAT: merging all shards into a single message (identical text, bullet-pointed) restores 95% of single-turn performance → the problem is the multi-turn format, not the content.
- RECAP: a final "summarize all requirements" prompt barely helps — the model is already sunk in earlier wrong assumptions.
- SNOWBALL: repeating all prior shards each turn improves things slightly but doesn't eliminate the drop → the model isn't forgetting information; it is misinterpreting it, and that misinterpretation has inertia.
- Temperature: lowering temperature does not improve multi-turn results → the flaw is in the model's reasoning strategy within multi-turn structure, not randomness.
- MemPrompt: stores user corrections and retrieves them into prompts for similar queries
- MemBART: dual attention streams for memory read/write
- Externalizing cross-turn information into explicit memory reduces reliance on implicit contextual reasoning.
- Vicuna: fine-tuned on real ChatGPT conversations preserving multi-turn traits
- UltraChat: self-dialogue multi-turn data (with scriptedness issues)
- Parrot: simulated human questioning plus negative samples (context ignoring/misunderstanding) for preference learning
- Gradient reweighting: Chen et al. (2025) found early-turn gradients cancel later-turn gradients; doubling last-two-turn gradient weights improves results.
- A name is not an explanation: the CONCAT experiment shows the problem is the *temporal* pattern of information exposure, not the information itself. Next-token prediction training inherently rewards continuity over reversal — models are trained to go along with prior text, not to overturn and restart.
- Cargo-cult detection: common "best practices" (prompt tweaks, lower temperature, summary turns) mimic what looks right without touching the root cause; the control experiments coldly show most are ineffective. Real solutions require architectural change, not surface tuning.
- Demonstration over argument: the paper's strength lies in 200K+ systematically controlled experiments, not theory or anecdotes.
- Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025). *LLMs Get Lost In Multi-Turn Conversation*. arXiv:2505.06120. ICLR 2026 Best Paper.
- Liu, G., et al. (2026). *Bridging the Intent Alignment Gap in Multi-Turn LLM Conversations*. arXiv:2602.07338.
- Liu, G., et al. (2026). *Contextual Inertia: The Root Cause of Multi-Turn Interaction Failures*. arXiv:2603.04783.
- Li, M. (2025). *Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation*. arXiv:2510.18731.
- Chen et al. (2025). *Addressing Multi-Round Gradient Cancellation in LLM Fine-Tuning*.
- Madaan et al. (2022). *MemPrompt*. Wu & Yu (2024). *MemBART*.
- ICLR 2026 coverage: https://www.jku.at/en/institute-for-symbolic-artificial-intelligence/news-events/detail/news/outstanding-paper-award-at-iclr-2026/
Example math problem:
This precisely simulates real scenarios where users don't know what they want upfront and reveal information gradually.
2.2 Key Data: Single-Turn vs Multi-Turn
| Model | Single-turn (FULL) | Multi-turn (SHARDED) | Drop | |---|---|---|---| | GPT-4.1 | 91.7% | 70.7% | -21.0% | | Claude 3.7 Sonnet | 85.4% | 70.0% | -15.4% | | Gemini 2.5 Pro | 90.2% | 64.3% | -25.9% | | DeepSeek-R1 | ~85% | ~60% | -25% | | Llama 3.1-8B | ~65% | ~45% | -20% | | Average | ~90% | ~65% | -39% |
Key insight: Even SOTA models perform only marginally better than small models in multi-turn settings — this is not a problem scale can solve.
2.3 Performance Decomposition: Aptitude vs Unreliability
1. Aptitude: best-case performance along optimal conversation paths 2. Unreliability: the gap between best and worst cases
The model's performance depends heavily on luck: whether it guessed the user's intent correctly in early turns.
3. Four Failure Modes: Why Models Can't Come Back from a Wrong Turn
1. Overly verbose: long responses add noise that the model struggles to extract key information from in later turns. 2. Premature final solutions (the most critical mode): the model jumps to an "answer" before information is complete; subsequent turns become patch-fixing instead of rethinking. 3. Incorrect assumptions: given missing information, the model fills in gaps (defaulting to SQLite, JSON, simplest edge cases) rather than asking — and cannot assess whether its assumptions are reliable. 4. Over-reliance on previous incorrect answers: once earlier turns produce wrong content, the model treats it as "confirmed fact" and incrementally patches rather than reasoning from scratch. Analogy: scribbling corrections over wrong code on a whiteboard instead of grabbing a clean whiteboard. The model has no "new whiteboard" ability.
4. Control Experiments: Ruling Out Alternative Explanations
5. Follow-Up Research and Improvement Directions
5.1 Mediator-Assistant architecture (Liu et al., 2026)
Decouples intent understanding from task execution: a Mediator consolidates vague multi-turn input into a clear single-turn instruction; an Assistant executes it. This significantly mitigates the drop — though one might ask whether an LLM-based Mediator could itself get lost in conversation.5.2 Contextual Inertia
Liu et al. (2026) find 70–90% of multi-turn errors trace back to propagation of earlier mistakes, and design mechanisms forcing models to "rethink" rather than patch when new information conflicts with old reasoning.5.3 Memory-augmented methods
5.4 SFT strategies
5.5 Verifiable accuracy and abstention rewards (Li, 2025)
Curriculum RL teaches models to abstain when information is insufficient instead of guessing blindly, using verifiable accuracy rewards instead of naive completion rewards.6. Practical Implications for Agent Development
Do now
1. Conversation splitting: start a new chat after 5–6 turns or a clear topic switch 2. Explicit summarization: at key checkpoints, have the agent state "Here is my understanding of the requirement..." to convert implicit context into explicit confirmation 3. Limit per-turn output length to reduce noise for later turnsArchitecture
1. Separate intent and execution layers 2. Explicit state management via external state machines/memory rather than implicit context 3. Self-checks: periodically ask "Do my assumptions still hold?" and re-evaluate against new informationEvaluation
1. Multi-turn benchmarks must become standard — strong single-turn performance doesn't mean real-world usability 2. Reliability metrics matter more than aptitude — agent users care about consistent results, not occasional perfection7. Deeper Questions
8. References
Closing thought: The paper's value isn't telling us "multi-turn is hard" — we knew that. It quantifies the difficulty (39%), identifies concrete failure modes (premature assumptions, over-reliance), and rules out pseudo-explanations with clever controls. For agent builders, this is a cold but necessary reminder: no matter how strong your model looks on benchmarks, it may perform at 60% in real multi-turn dialogue. Design your agent assuming it will stubbornly get lost, and build the system around that assumption.
*Source: arXiv:2505.06120, ICLR 2026 Best Paper*