ICLR 2026 Best Paper: Why LLMs Lose 39% Performance in Multi-Turn Conversations

> Key takeaway: All major LLMs (from Llama 3.1-8B to GPT-4.1 and Gemini 2.5 Pro) show an average 39% performance drop in multi-turn conversation. The issue is not that models become "dumber," but that reliability collapses: once a model commits to a wrong assumption early in the dialogue, it slides into a hole it cannot climb out of.

---

1. Paper at a glance

| Attribute | Detail | |---|---| | Title | LLMs Get Lost In Multi-Turn Conversation | | Authors | Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville | | Affiliations | Microsoft Research, Salesforce Research | | Venue | ICLR 2026 (Best Paper Award) | | arXiv | 2505.06120 | | Scope | 15 LLMs, 200,000+ dialogues, 6 generation tasks |

---

2. The 39% drop, decomposed

2.1 Methodology: Shard Simulation

The authors split a single complete instruction into a sequence of partial turns, mirroring real users who only know what they want incrementally.

• Fully-Specified (single turn): every constraint delivered at once.
• Sharded (multi-turn): the same information released one piece per turn.

Example (math word problem):

• Single turn: "Jay makes 20 snowballs per hour, but 2 melt every 15 minutes. How long to reach 60?"
• Multi-turn:
• Turn 1: "Jay is making snowballs for a snowball fight."
• Turn 2: "He can make 20 per hour."
• Turn 3: "But 2 melt every 15 minutes."
• Turn 4: "He needs 60. How long?"

2.2 Headline numbers

| Model | Single-turn (FULL) | Multi-turn (SHARDED) | Drop | |---|---|---|---| | GPT-4.1 | 91.7% | 70.7% | -21.0% | | Claude 3.7 Sonnet | 85.4% | 70.0% | -15.4% | | Gemini 2.5 Pro | 90.2% | 64.3% | -25.9% | | DeepSeek-R1 | ~85% | ~60% | -25% | | Llama 3.1-8B | ~65% | ~45% | -20% | | Average | ~90% | ~65% | -39% |

Even SOTA models are only marginally better than small ones in multi-turn settings. Scale does not fix this problem.

2.3 Aptitude vs. unreliability

• Aptitude: best-case performance along an optimal conversational path. Strong models do well here, but only in single-turn settings.
• Unreliability: gap between best and worst cases. In multi-turn, every model shows a huge gap, meaning the same model on the same task can succeed or fail depending purely on the path the user happened to take.

> Strong single-turn benchmark scores do not imply understanding. Multi-turn evaluation reveals that model behavior depends heavily on luck, specifically whether the model guessed the user's intent correctly in the first turn.

---

3. Four failure modes

3.1 Overly verbose

Long responses bury key information in noise that later turns struggle to parse.

3.2 Premature final solutions (the core failure)

Models answer before the picture is complete, then all subsequent turns become "patch the wrong answer" instead of "rethink from scratch."

3.3 Incorrect assumptions

Models never say "I don't know" or "please clarify." They hallucinate defaults (SQLite, JSON, simplest edge case), and have no internal mechanism to assess whether those assumptions are sound.

3.4 Over-reliance on previous incorrect answers (context contamination)

Once a wrong statement is in the context, the model treats it as confirmed fact. New information is used to patch the old answer rather than trigger a fresh derivation.

> Analogy: writing wrong code on a whiteboard and then trying to overwrite it with a marker, instead of erasing the board and starting over. LLMs have no "erase" operation.

---

4. Control experiments rule out easy explanations

4.1 CONCAT — content is not the problem

Joining all multi-turn fragments into a single bullet-list message restores performance to ~95% of single-turn.

> The information is identical; only the turn-by-turn format hurts.

4.2 RECAP — final reminders do not help

Adding a turn that summarizes every requirement has minimal effect.

4.3 SNOWBALL — repeating past fragments does not help

Re-injecting prior fragments each turn yields only small gains.

> The model is not forgetting. It is misinterpreting, and the misinterpretation has inertia.

4.4 Temperature sweep — randomness is not the problem

Lowering temperature does not fix the drop.

> The defect lives in the model's multi-turn reasoning strategy itself, not in sampling noise.

---

5. Follow-up research and improvement directions

5.1 Mediator-Assistant architecture (Liu et al., 2026)

Decouple intent understanding from task execution:
• Mediator: turns a fuzzy, multi-turn dialog into a clean, single-turn-style instruction.
• Assistant: executes the clarified instruction.

Significantly mitigates the drop in multi-turn settings, though the Mediator is itself an LLM and may inherit the same pathology.

5.2 Contextual Inertia (Liu et al., 2026)

• 70–90% of multi-turn errors trace back to propagation of earlier-turn mistakes.
• Proposes mechanisms that force the model to "rethink" rather than "patch" when new information conflicts with prior reasoning.

5.3 Memory-augmented methods

• MemPrompt (Madaan et al., 2022): stores user corrections and retrieves them for similar future queries.
• MemBART (Wu & Yu, 2024): dual attention over memory read and write streams.
• Externalizing cross-turn state reduces the burden on implicit in-context reasoning.

5.4 SFT strategies

• Vicuna: fine-tune on real ChatGPT conversations.
• UltraChat: self-chat generated multi-turn data (with scripted artifacts).
• Parrot: human-style questioning plus negative samples (ignored/misread context) for preference learning.
• Gradient weighting (Chen et al., 2025): early-turn gradients can cancel later-turn gradients; doubling the weight on the last two turns improves results.

5.5 Verifiable Accuracy & Abstention Rewards (Li, 2025)

Curriculum RL that rewards:
• Abstention when information is insufficient, instead of blind guessing.
• Verifiable accuracy rather than generic completion likelihood.

---

6. Practical implications for agent development

6.1 Immediate tactics

1. Reset context when a dialog exceeds ~5–6 turns or the topic shifts. 2. Explicit confirmation checkpoints ("Here is what I understand you want: ...") to convert implicit context into explicit agreement. 3. Cap response length to limit information noise in later turns.

6.2 Architectural changes

1. Separate the intent-understanding layer from the task-execution layer. 2. Use an external state machine or memory store; do not rely on the LLM's implicit context. 3. Add self-check routines ("Are my assumptions still valid?") that trigger re-derivation when needed.

6.3 Evaluation changes

1. Multi-turn benchmarks must become standard. Single-turn accuracy is a poor proxy for real usability. 2. Reliability metrics matter more than peak capability for production agents. Users prefer consistent results over occasional brilliance.

---

7. Deeper questions

Naming is not understanding

The phrase "Lost in Conversation" points at dialogue, but the CONCAT experiment shows identical text in single-turn form works fine. The real culprit is the temporal exposure of information. Humans can "erase the whiteboard" on hearing new information; next-token training encourages continuity, not reversal.

Cargo-cult detection

Common workarounds (prompt tricks, lower temperature, summary turns) do not move the needle. Genuine progress requires architectural change, not surface tuning.

Empirical strength

The paper's authority comes from systematic, large-scale controlled experiments: 200,000+ dialogues, not anecdotal case studies.

---

8. References

• Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025). *LLMs Get Lost In Multi-Turn Conversation*. arXiv:2505.06120. ICLR 2026 Best Paper.
• Liu, G., et al. (2026). *Bridging the Intent Alignment Gap in Multi-Turn LLM Conversations*. arXiv:2602.07338.
• Liu, G., et al. (2026). *Contextual Inertia: The Root Cause of Multi-Turn Interaction Failures*. arXiv:2603.04783.
• Li, M. (2025). *Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation*. arXiv:2510.18731.
• Chen et al. (2025). *Addressing Multi-Round Gradient Cancellation in LLM Fine-Tuning*.
• Madaan et al. (2022). *MemPrompt*.
• Wu & Yu (2024). *MemBART*.
• ICLR 2026 award page: https://www.jku.at/en/institute-for-symbolic-artificial-intelligence/news-events/detail/news/outstanding-paper-award-at-iclr-2026/

> Final note: The paper's value is not the observation that multi-turn dialog is hard. It is the quantification (39%), the taxonomy of failure modes (premature answers, over-reliance on prior errors), and the disciplined control experiments that rule out false explanations. For agent builders, the sober lesson is: assume your model will stubbornly get lost, then design the system around that assumption.