> Have you ever met someone like this — you ask "where should I go this weekend" and they instantly list ten attractions; by the time you add "my budget is only 200 yuan and I don't want to hike," they've already spent half an hour on mountain-climbing guides. Split the conditions across a few sentences, and it's as if they can't hear the second half. But if you stack every condition into one paragraph up front, they answer perfectly. Same brain, two input styles, wildly different results. This isn't a story about people. It's the daily reality of nearly every language model today.
---
| Item | Detail | |------|--------| | Paper | Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap | | Authors | Tianlang Chen, Shirley Wu, Jure Leskovec | | Institution | Stanford University | | arXiv ID | 2605.24432 | | Submitted | May 23, 2026 | | Category | cs.CL (Computation and Language) | | Key finding | LLM performance drops ~39% on average in multi-turn underspecified conversation (Lost-in-Conversation); the FiC framework — SFT warm-start + View-Asymmetric Self-Distillation — restores at least 92% of single-turn performance across Llama/Qwen/Phi/OLMo (3B–14B) without an external teacher, reaching 100% on Llama backbones, with 33% better token efficiency |
1. An Overlooked Everyday Fracture
In the real world, people rarely say everything at once. You open a chat window, toss out a vague request, the other side asks a few questions, you add details — three or four turns later the task is done. That's natural human communication.
LLMs don't see it that way. Pack all the conditions into one long prompt and it does fine. Split the same conditions over three turns and performance can collapse by a third. The gap is not a capability problem — it's a behavioral problem. The model has the skill inside; it just can't deploy it when information arrives in batches.
This phenomenon is called Lost-in-Conversation. Work by Laban et al. (2025) found fifteen mainstream models (open and closed, large and small) dropping 39% of task scores on average in multi-turn underspecified settings. Give a math word problem in one shot and the model solves it; give it in three turns and it may eagerly commit to a wrong assumption in turn one, anchoring on it and drifting further off course afterward.
Chen, Wu, and Leskovec at Stanford asked a direct question: if the same model can already succeed in the single-turn setting, the competence is clearly there — can we get the model to teach itself, carrying single-turn competence into multi-turn settings?
2. Three Information-Equivalent Doors
The authors designed a careful controlled comparison: a single-turn instruction presented in three information-equivalent ways.
1. FULL — the original single-turn instruction with all conditions at once. Baseline; best performance. 2. CONCAT — same single-turn structure, but conditions listed as a bullet list. Information content identical to FULL, only the formatting changes. Tests whether fragmentation alone hurts. 3. SHARDED — true multi-turn: first condition in turn one, second in turn two, and so on. Each turn alone is insufficient; only together do they match FULL. This is where Lost-in-Conversation happens.
FULL and CONCAT perform essentially the same. So the model isn't degraded by "information chopped into small pieces" — it reads bullet lists just fine. The real problem lies on the time axis: when information arrives in batches, what choices does the model make in the early turns?
3. The Premature Commitment Trap
The culprit is what the authors call premature commitment. Given incomplete first-turn information, the model doesn't say "I'd also need X and Y to answer" — it instinctively fills the gap with a hallucinated assumption and produces a plausible-looking answer. Worse, once stated, that assumption becomes part of the context for later turns: the model reasons on top of its own mistake and sinks deeper.
Where does this come from? Likely standard post-training objectives: models are trained to maximize immediate next-turn reward — to answer quickly and satisfy the user. That objective is toxic when information is incomplete: it rewards rushing and penalizes careful deferral.
The first stage of FiC targets exactly this. An SFT warm-start on carefully constructed multi-turn corpora trains the model to defer answering when information is insufficient, requesting the specific missing details until everything necessary has arrived.
The corpus construction is meticulous. Starting from a standard single-turn benchmark (e.g., GSM8K math problems), an LLM splits each problem's full conditions into information shards — each insufficient alone, jointly equivalent to the original. A validation step then reassembles the shards into a CONCAT bullet list and confirms performance matches FULL; only shard sets passing this check enter training.
The design of intermediate-turn gold answers is where the craft shows. Rather than a fixed "please provide more info" template, the authors diversify along two dimensions:
- Style diversity: deferral phrasing varies in format, wording, register, and length across samples and turns, so the model learns the *behavior* of deferral, not one specific sentence.
- Position-aware specificity: in early turns, with extremely sparse context, the model cannot know what's missing, so gold answers are generic deferrals ("I need more information"). In later turns, with richer context, gold answers name the missing pieces ("I still need the speed and the distance") — more useful to the user. Every intermediate answer passes a judge that verifies it requests genuinely missing and necessary information, preventing learned stonewalling.
- Families: Llama, Qwen, Phi, OLMo, sizes 3B–14B. On Math, FiC recovers at least 92% of single-turn performance for every model; two Llama backbones reach 100% — multi-turn fully matching single-turn.
- Single-turn capability is essentially unharmed after training — no sawing off the strong leg to fix the weak one.
- A bonus: models post-FiC use 33% fewer tokens in multi-turn conversation. Why? They no longer ramble when information is incomplete — one crisp "I still need X" and they stop. Shorter, more direct, cheaper conversations.
- Cross-domain transfer: trained only on Math, FiC checkpoints transfer to Database (SQL generation) and Actions (API calling), recovering most of the gap with less than 6 A100-hours of adaptation. This suggests FiC teaches a task-agnostic meta-strategy: don't commit prematurely; assess whether current information suffices; if not, request more — a way of thinking, not a specific fact.
- The CONCAT control convinced me the problem is behavioral, not a tokenization/attention artifact of chat templates: same fragmented information given all at once restores performance, so the issue is what the model *does* when information arrives in batches.
- "Self-distillation" sounded like repackaged hype at first, but the real novelty is clean: teacher and student are information-equivalent, differing only in temporal structure — not big-teaches-small knowledge transfer, but behavioral alignment of the same knowledge under different time layouts.
- On-policy sampling worries me: the training signal partly rests on the student's own possibly-bad intermediate answers. Can teacher alignment pull back a truly off-the-rails early turn? No failure-case analysis in the reviewed pages.
- Path A: prompt engineering ("if information is incomplete, ask first"). Cheapest, lowest ceiling; hard to overturn entrenched behavior, highly prompt-sensitive.
- Path B: RL with sparse rewards. Credit assignment over long dialogues is hard; sparse rewards propagate poorly across long trajectories — a pitfall the authors explicitly avoid.
- Path C: external teacher distillation. Even frontier models have the multi-turn gap; hiring a nearsighted eye doctor has limited effect.
- Path D: data augmentation with real multi-turn dialogues. "Good multi-turn conversations" are hard to define and annotate; real distributions are skewed.
The final-turn gold answer isn't human-written — it comes from the model's own correct generations in the CONCAT view. This keeps the SFT supervision close to the model's own output distribution, avoiding the distribution shift of "teaching yourself with someone else's words." Multiple correct CONCAT rollouts per problem are sampled randomly each epoch, further preventing collapse onto a single canonical phrasing.
4. Two Selves in the Same Mirror
SFT cures the rush-to-answer problem, but multi-turn accuracy still trails single-turn. The remaining gap is closed by View-Asymmetric Self-Distillation (VASD): run the same model in two views — the single-turn CONCAT view as teacher, the multi-turn SHARDED view as student — and pull the student's token-level distribution toward the teacher's at the final answer position using Jensen-Shannon divergence.
Key design points:
1. Same backbone for teacher and student. No external stronger model is needed, because frontier models share the same gap — hiring a nearsighted optometrist makes little sense. 2. Information equivalence. The teacher sees CONCAT (full information as a bullet list); the student sees SHARDED (full information arriving in batches). By the last turn both hold the identical condition set. Any divergence between their distributions therefore comes purely from the input view's temporal structure, not the content — an extremely clean learning signal. 3. On-policy intermediate sampling. The student's intermediate-turn responses are its own generations, and the rollout continues from them. This avoids the off-policy mismatch of "training on someone else's drafts, tested writing your own."
Jensen-Shannon divergence rather than KL is a notable choice: JS is symmetric and gentler on extreme probabilities on either side, less prone to blowing up on outliers. Whether the paper ablates JS vs. KL isn't clear from the pages reviewed here.
5. The Numbers
FiC was validated on Laban et al.'s multi-turn benchmark across families and scales.
Honest caveat: the reviewed pages don't give exact recovery percentages for Database and Actions — only "most of the gap." Whether that's 85% or 95% remains unclear.
6. Points of Hesitation
7. A Signal-Processing Reframe
A model in multi-turn dialogue maintains a belief distribution over time, updated by each user turn. Ideally the update is sufficient and necessary: absorb task-relevant information, ignore noise, and preserve uncertainty when information is scarce rather than collapsing to a guess.
Lost-in-Conversation is premature belief collapse: pushed by the optimization objective, the model converges to a low-confidence point estimate instead of maintaining a broad posterior, and subsequent conditioning on that guess makes reversal hard.
FiC's two stages map to classic signal-processing strategies: the SFT warm-start is a threshold detector at the front end — when input SNR is too low, output "refuse to decode" rather than guess; VASD is a distribution aligner at the back end — ensuring the same signal converges to the same steady-state output distribution regardless of arrival timing. The 33% token reduction fits too: a prematurely collapsed model generates verbose explanations as rhetorical smoke screens, while a model that learned to defer emits a high-confidence request-for-more signal with no detours. (This reconstruction is my own interpretation, not necessarily the authors'.)
8. Alternative Roads Not Taken
9. Open Questions
1. Why is CONCAT a valid teacher? A bullet list may itself be a crutch; real users write prose. Would a prose teacher degrade? Untested in the reviewed pages. 2. On-policy sampling stability: can low-quality early-turn generations derail training? No training dynamics shown. 3. The cost of 33% fewer tokens: shorter history per turn — does less context hurt adaptability when users change course? An efficiency/flexibility trade-off is undiscussed. 4. Is Llama's 100% recovery an outlier? Others recover 92–98%. Architecture or pretraining data? Unexplained. 5. RLVR synergy: are distillation-based FiC and verifiable-reward RL mutually exclusive or complementary? 6. User simulator fidelity: evaluation and training both depend on an LLM-based simulator; real users are far more diverse. Robustness beyond the simulator is an open ecosystem question. 7. SFT vs. VASD contributions: no ablation in the reviewed pages — additive, multiplicative, or is SFT a precondition for VASD to converge? 8. Scaling: does recovery rise monotonically with size? Untested beyond 14B.
10. What "Teach Themselves" Actually Means
The phrase "LLMs Teach Themselves" invites headlines about self-evolving AI, but the reality is narrower and more precise. The model learns no new knowledge — its parameters gain nothing. It learns to schedule existing knowledge when information arrives in batches. The single-turn teacher could already solve the problem; the multi-turn student couldn't, not for lack of knowledge, but because the temporal layout of information disrupted its behavior. FiC is behavior correction, not knowledge injection.
A piano analogy: a pianist who knows a piece perfectly may play it badly if told one hand's notes at a time, with the other hand's position arriving later. Not because they can't play — the staggered perception disrupts their motor program. FiC is a training regimen that lets the same performance program execute under staggered information. The analogy's limits are obvious — models have no fingers or muscle memory — but the structure matches: temporal interference degrading an already-present capability.
11. Paper Metadata
| Item | Detail | |------|--------| | Title | Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap | | Authors | Tianlang Chen, Shirley Wu, Jure Leskovec | | Institution | Stanford University | | arXiv ID | 2605.24432 | | Submitted | May 23, 2026 | | Category | cs.CL | | Method | Two stages: SFT warm-start (learn to defer) + View-Asymmetric Self-Distillation (CONCAT teacher → SHARDED student, token-level JS divergence) | | Benchmarks | Laban et al. [2025] multi-turn benchmark: Math (GSM8K), Database (Spider), Actions (BFCL) | | Views | FULL (original single-turn), CONCAT (bullet-list single-turn), SHARDED (multi-turn fragments) | | Models tested | Llama, Qwen, Phi, OLMo (3B–14B) | | Main results | ≥92% single-turn recovery across families (100% on Llama); 33% token efficiency gain; cross-domain transfer to Database/Actions (<6 A100 hours) | | Core insight | The multi-turn gap is behavioral, not capability; a model's own single-turn view can serve as a teacher for its multi-turn view; on-policy sampling + information-equivalent views yield a clean alignment signal |
*(Translator's note: this is an editorial English rendering of a Chinese forum post; opinions and analogies in sections 6–10 belong to the original author.)*