LLM4CS: A Prompting Framework Using Large Language Models for Conversational Search

Forum topic · 小凯 · 2026-07-05

Summary

LLM4CS is a simple yet effective prompting framework that leverages large language models (LLMs) as text-based search intent interpreters for conversational search. Understanding users' contextual search intent remains a core challenge in conversational search, where sessions are diverse and long-tailed, leaving supervised methods trained on limited data brittle in real-world scenarios. LLM4CS explores three prompting strategies that prompt an LLM to generate multiple query rewrites and hypothetical responses for each conversation turn; these outputs are then aggregated into an integrated representation that robustly captures the user's true contextual intent. Extensive automatic and human evaluations on three widely used benchmarks—CAsT-19, CAsT-20, and CAsT-21—show that LLM4CS outperforms existing methods and even surpasses pipelines built on human-issued rewrites. The work, authored by researchers including Kelong Mao and Zhicheng Dou (arXiv:2303.06573, March 2023), offers important evidence on how to understand and exploit LLMs for multi-turn information seeking without task-specific training.

LLM4CS: Large Language Models Know Your Contextual Search Intent

Paper: Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search Authors: Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, Hongjin Qian Published: 2023-03-12 (arXiv) Topics: Conversational Search, Multi-Turn Retrieval

Problem

Precisely understanding users' contextual search intent has been an important challenge for conversational search. Conversational search sessions are much more diverse and long-tailed, so existing methods trained on limited data still show unsatisfactory effectiveness and robustness in real conversational search scenarios. Meanwhile, large language models (LLMs) have demonstrated impressive capabilities in text generation and conversation understanding.

The LLM4CS Framework

The authors present a simple yet effective prompting framework, LLM4CS, which uses LLMs as a text-based search intent interpreter to assist conversational search — with no task-specific fine-tuning required.

Key components:

1. Prompting strategies — Three prompting methods elicit an LLM to interpret the conversation context, generating:

Multiple query rewrites (self-contained reformulations of the current turn)
Hypothetical responses (anticipated answers that reveal the underlying intent)

Aggregation

Retrieval

Results

Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks — CAsT-19, CAsT-20, and CAsT-21 — demonstrate that the simple LLM4CS framework achieves remarkable performance compared with existing methods, and even outperforms pipelines that rely on human-issued query rewrites.

Why It Matters

Provides strong evidence that LLMs implicitly "know" a user's contextual search intent and can surface it through careful prompting.
Offers a training-free alternative to supervised conversational query reformulation, which generalizes poorly to long-tailed sessions.
Establishes query rewriting + hypothetical response generation + aggregation as a reusable recipe for multi-turn retrieval.

Original Abstract (English)

> Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.