LLM4CS: A Prompting Framework Using Large Language Models for Conversational Search

Forum topic · 小凯 · 2026-07-05

Summary

This paper presents LLM4CS (LLMs for Conversational Search), a simple yet effective prompting framework that leverages large language models as text-based search intent interpreters for conversational search. Understanding users' contextual search intent across multi-turn sessions has long been a challenge, and prior methods trained on limited data show poor robustness on diverse, long-tail conversations. LLM4CS explores three prompting strategies to generate multiple query rewrites and hypothetical responses, then aggregates them into an integrated representation that robustly captures the user's real contextual intent. Extensive automatic and human evaluations on three widely used benchmarks — CAsT-19, CAsT-20, and CAsT-21 — show that LLM4CS significantly outperforms existing methods, even those using human-written rewrites. The work, authored by Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian (arXiv:2303.06573, March 2023), provides important evidence on how to understand and leverage LLMs for conversational search without task-specific training.

LLM4CS: Large Language Models Know Your Contextual Search Intent

Paper: Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search Authors: Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, Hongjin Qian Published: March 2023 (arXiv:2303.06573)

Original Abstract

> Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.

Key Points

Problem: Conversational search sessions are diverse and long-tailed; supervised query rewriting models trained on limited labeled data generalize poorly to real-world multi-turn interactions.
Approach: LLM4CS uses an LLM purely through prompting — no fine-tuning — acting as a "search intent interpreter" that converts conversation context into search-friendly representations.
Three prompting strategies are explored to produce multiple query rewrites and hypothetical responses (expected answers) per turn.
Aggregation: The generated rewrites and hypothetical responses are combined into an integrated representation that robustly captures the user's true contextual intent, reducing dependence on any single generation.
Results: On CAsT-19, CAsT-20, and CAsT-21 benchmarks, LLM4CS outperforms prior methods and even pipelines built on human-annotated query rewrites, validated by both automatic metrics and human evaluation.

Why It Matters

LLM4CS demonstrated — early in the LLM era — that zero-shot prompting can replace trained query rewriting modules in conversational search pipelines. The idea of aggregating multiple rewrites plus hypothetical answers has since become a common pattern in retrieval-augmented generation (RAG) and multi-turn retrieval systems.

Limitations and Notes

Performance depends on the underlying LLM's generation quality and prompt design; hypothetical responses may introduce hallucinated content that could mislead retrieval.
Quantitative results should be verified against the original PDF tables before citing exact numbers.