Proactive Guidance of Multi-Turn Conversation in Industrial Search
This post summarizes the arXiv paper Proactive Guidance of Multi-Turn Conversation in Industrial Search (May 30, 2025), by Xiaoyu Li, Xiao Li, Li Gao, Yiding Liu, Xiaoyang Wang, Shuaiqiang Wang, et al. (8 authors), deployed in the Baidu Search AI assistant.
Key points
- Problem: LLM-based multi-turn conversation systems must dynamically adapt to shifts in user goals while maintaining low latency for real-time interactions — a challenge for industrial-scale search.
- Two-phase framework: 1. Goal-adaptive Supervised Fine-Tuning (G-SFT): a goal adaptation agent dynamically tracks user goal shifts and provides goal-relevant context; scalable knowledge transfer distills LLM insights into a lightweight model for real-time serving. 2. Click-oriented Reinforcement Learning (C-RL): a generate-rank paradigm that systematically builds preference pairs from user click signals and improves CTR through more engaging proactive guidance.
- Results:
- Offline accuracy: 86.10% (+23.95% over baseline)
- Online CTR: 25.28% (149.06% relative improvement)
- Inference latency reduced by 69.55% via scalable knowledge distillation
- The work sits at the intersection of agentic search and large-scale search/recommendation systems, rethinking the division of labor among retrieval, ranking, generation, and tool calling in the LLM era.
- Architecturally, cascaded retrieve-rank-generate remains mainstream, but agentic paradigms treat retrieval count and strategy as learnable decisions.
- Both high-quality instruction data and click/session logs matter; synthetic data requires safeguards against knowledge leakage and distribution shift.
- The offline-online gap continues to widen; LLM-as-judge evaluation should be cross-validated with human assessment.
- Latency, cost, interpretability, and safety are hard constraints for industrial deployment — academic benchmarks alone are insufficient.
- A Survey on Multi-Turn Interaction Capabilities of Large Language Models
- Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
- Beyond Whole Dialogue Modeling: Contextual Disentanglement for Conversation
- CHIQ: Contextual History Enhancement for Improving Query Rewriting
- Original paper: Proactive Guidance of Multi-Turn Conversation in Industrial Search
Context and insights
Engineering checklist (from the post)
| Area | Concern | Suggestion | |------|---------|------------| | Data | PII, version management | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | p99 budget, retrieval depth | Cascades + early stopping, caching hot queries, async reranking | | Quality | Offline gains to online CTR | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias from open retrieval | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token/GPU spend | Route to smaller models, distillation, hybrid sparse+dense retrieval |