LLM4CS: Large Language Models Know Your Contextual Search Intent
Overview
Paper: Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search
Authors: Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, Hongjin Qian
Published: 2023-03-12
Link: https://arxiv.org/abs/2303.06573
Problem
Precisely understanding users' contextual search intent has been an important challenge for conversational search. Conversational search sessions are much more diverse and long-tailed, so existing methods trained on limited supervised data show unsatisfactory effectiveness and robustness in real scenarios. Meanwhile, large language models (LLMs) have demonstrated strong capabilities in text generation and conversation understanding.
The LLM4CS Framework
LLM4CS (LLM for Conversational Search) uses LLMs as a text-based search intent interpreter:
1. Prompting methods: Three prompting strategies are explored to elicit from the LLM multiple query rewrites and hypothetical responses for each conversation turn. 2. Aggregation: The generated rewrites and responses are aggregated into an integrated representation that robustly captures the user's real contextual search intent. 3. Retrieval: The aggregated representation is used for retrieval in the multi-turn search pipeline.
Results
Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks — CAsT-19, CAsT-20, and CAsT-21 — demonstrate the remarkable performance of LLM4CS compared with existing methods and even with pipelines using human query rewrites. The findings provide important evidence for better understanding and leveraging LLMs in conversational search.
Key Takeaways
- LLMs can act as zero-shot intent interpreters without task-specific training data.
- Generating and aggregating multiple interpretations (rewrites + hypothetical responses) is more robust than relying on a single rewrite.
- Prompting-based intent modeling can exceed human-rewrite baselines on CAsT benchmarks.
- A Survey of Conversational Search (Sep 2025, ACM)
- Engineering Conversational Search Systems: A Review (arXiv:2407.00997)
- ChatRetriever: Adapting LLMs for Generalized and Robust Conversational Search (arXiv:2404.13556)
- CoSearchAgent: A Lightweight Collaborative Search Agent (arXiv:2402.06360)
- Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
Original Abstract (verbatim)
> Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.