Overview
This report analyzes the ICLR 2026 Best Paper "LLMs Get Lost In Multi-Turn Conversation" (arXiv:2505.06120) by Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville (Microsoft Research, Salesforce Research). ICLR reviewers gave perfect scores (Soundness 4/4, Presentation 4/4, Contribution 4/4, Rating 10/10) and called it a high-impact paper exposing a major flaw in current evaluations.
Key headline numbers
| Metric | Value | |---|---| | Models tested | 15 (open + closed frontier) | | Simulated dialogues | 200,000+ | | Domains | 6 (code, DB, actions, math, summarization, NLG) | | Single-turn accuracy | ~90% | | Multi-turn accuracy | ~65% | | Performance drop | ~39% | | Onset | Significant from turn 2 onward |
A counter-intuitive finding: the drop is driven less by loss of aptitude than by a large spike in unreliability (run-to-run variance). Temperature 0.0 reduces unreliability only marginally (~30 percentage points remain), so the failure is structural rather than stochastic.
Sharded Simulation methodology
Single-turn benchmarks systematically overestimate real-world capability because users provide information incrementally. The authors introduce Sharded Simulation: an atomic-content-unit extraction step rewrites a full instruction into underspecified turns, e.g.:
- Turn 1: "Jay is making snowballs."
- Turn 2: "He can make 20 per hour."
- Turn 3: "But every 15 minutes, 2 melt."
- Turn 4: "He wants 60. How long?"
- Mediator-Assistant architecture (arXiv:2602.07338v1) — splits the system into an intent Mediator and an execution Assistant to resolve intent mismatch before task execution.
- RLAAR (arXiv:2510.18731, UMD + Amazon) — Curriculum RL with dual rewards for accuracy and calibrated abstention. Reported LiC accuracy improved from 62.6% to 75.1%, and calibrated abstention rose from 33.5% to 73.4%.
- Memory augmentation survey (arXiv:2504.04717v2) — covers MemPrompt, MemBART, RAG, knowledge graphs, and dynamic context compression as "external hard drives" for conversation state.
- SFT-only fixes are insufficient — even Vicuna and ChatGLM2, trained on multi-turn data, still exhibit strong Lost-in-Conversation effects.
- For LLM builders: optimize multi-turn reliability, not just single-turn ability. Target U90-10 (90th–10th percentile gap) under 15 points at temperature=1.0.
- For agent developers: do not assume LLMs remember multi-turn state even at 128K context. Use dynamic recap, structured state stores, and treat recap/snippet fixes as partial.
- For end users: when a conversation drifts, start a new chat and consolidate requirements into one prompt, or explicitly request a recap.
- arXiv:2505.06120 — LLMs Get Lost In Multi-Turn Conversation (ICLR 2026 Best Paper)
- arXiv:2510.18731 — RLAAR
- arXiv:2602.07338v1 — Mediator-Assistant
- arXiv:2504.04717v2 — Multi-turn interaction survey
- https://iclr.pangram.com/reviews
- https://www.microsoft.com/en-us/research/event/iclr-2026/
- https://www.getmaxim.ai/blog/from-turn-1-to-turn-10-how-llms-get-lost-in-multi-turn-conversations/
Five simulation modes isolate the cause:
| Mode | Description | |---|---| | FULL | Original single-turn instruction (baseline) | | SHARDED | Multi-turn incremental shards | | CONCAT | All shards merged back into one prompt | | RECAP | SHARDED plus a final summary turn | | SNOWBALL | Each turn repeats all prior shards plus a new one |
Because CONCAT matches FULL while SHARDED collapses, the cause is multi-turn structure itself, not paraphrase.
Four recurring failure modes
1. Premature Answer Attempts — the model produces a full answer on turn 1 despite missing information. It defaults to "guess first, correct later." 2. Over-Reliance on Their Own Mistakes / Answer Bloat — once an early guess is wrong, later turns patch and inflate it instead of retracting. Final answers in SHARDED runs are 20–300% longer than in FULL runs. 3. Forgetting Middle Turns / Loss-of-Middle-Turns — a multi-turn variant of the "Lost in the Middle" attention bias. In an 8-turn summarization task, turn 1 content was cited 96% of the time, turn 8 ~20%, and turns 2–3 only ~8%. 4. Verbosity Breeds Assumptions — longer model outputs correlate with worse downstream performance because they introduce unrequested assumptions that become context noise.
These four modes form a self-reinforcing loop: premature answers seed bloated errors, which consume attention and bury middle turns, which forces more verbosity and more assumptions.