LLM4CS: Using Large Language Models to Interpret Contextual Search Intent in Conversational Search

• Paper: Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search
• Authors: Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, Hongjin Qian
• Published: 2023-03-12
• Source: https://arxiv.org/abs/2303.06573
• Type: Academic paper — Conversational Search / Multi-Turn

One-sentence Summary

LLM4CS is a simple yet effective prompting framework that treats large language models as text-based search intent interpreters, aggregating multiple query rewrites and hypothetical responses to robustly capture a user's contextual intent in conversational search.

Background and Motivation

Understanding a user's contextual search intent has long been a central challenge in conversational search. Conversational sessions are far more diverse and long-tailed than standalone queries, and existing methods trained on limited data show unsatisfactory effectiveness and robustness in real scenarios. Meanwhile, large language models (LLMs) have demonstrated remarkable capabilities in text generation and conversation understanding, motivating their use for intent interpretation. The paper positions itself at the intersection of retrieval-augmented generation (RAG) and large-scale search systems, where the division of responsibilities among retrieval, ranking, generation, and tool calling must be redesigned for the LLM era.

The LLM4CS Framework

LLM4CS uses LLMs as text-based search intent interpreters:

1. Prompting for intent: Three prompting methods elicit multiple query rewrites and hypothetical responses from an LLM, capturing different interpretations of the conversational context. 2. Aggregation: The generated rewrites and hypothetical answers are aggregated into an integrated representation that robustly reflects the user's real contextual search intent. 3. Retrieval: The aggregated representation is used for downstream document retrieval in multi-turn sessions.

This follows the general pipeline of problem formalization → system design → construction → inference, with inputs encoded as structured prompts rather than trained conversational query reformulators.

Experiments and Evaluation

• Benchmarks: Three widely used conversational search datasets — CAsT-19, CAsT-20, and CAsT-21.
• Evaluation: Both extensive automatic evaluations and human evaluations.
• Findings: LLM4CS achieves remarkable performance compared with existing methods and even outperforms systems that rely on human rewrites, providing important evidence for understanding and leveraging LLMs in conversational search.

Specific numerical results should be verified against the original paper's tables.

Key Insights for Search and Personalization

1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms increasingly treat retrieval count and strategy as learnable decisions. 2. Data: High-quality instruction data and session/click logs are critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human assessment. 4. Productization: Latency, cost, explainability, and safety are hard constraints in industry and cannot be sacrificed for benchmark gains.

Limitations and Future Directions

Potential limitations include experiment scale constrained by compute budgets, mismatch between benchmarks and real user distributions, unknown cross-lingual generalization, and safety risks of agentic systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.

Related Entries

• A Survey of Conversational Search (ACM, Sep 2025)
• Engineering Conversational Search Systems: A Review of Applications (arXiv:2407.00997)
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
• CTR-Guided Generative Query Suggestion in Conversational Search (EMNLP)
• ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Search (arXiv:2404.13556)
• CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models (arXiv:2402.06360)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |

Original Abstract

> Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.