Aligning Query Representation with Rewritten Query and Relevance Judgments in Conversational Search (CIKM 2024)
This post is a forum-indexed entry for a CIKM 2024 paper on conversational search.
Metadata
| Field | Value | |-------|-------| | Title | Aligning Query Representation with Rewritten Query and Relevance Judgments in Conversational Search | | Venue | CIKM 2024 | | Source link | https://dl.acm.org/doi/abs/10.1145/3627673.3679534 | | Category | Multi-Turn Conversational Search |
One-line Summary
The work focuses on aligning query representations with rewritten queries and relevance judgments in conversational search.
Background and Motivation
In large-scale search, recommendation, and personalization systems, conversational search faces challenges in efficiency, scalability, and user intent understanding. Traditional pipelined approaches tend to separate retrieval, ranking, and generation, making it hard to meet LLM-era demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. This paper is positioned at this intersection, aiming to advance the theory and practice of conversational query understanding.
Core problem scenarios include: open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation, and end-to-end architectures combining external knowledge sources with generative models.
Key Contributions (as framed by the post)
- Provides a unified perspective that organizes scattered related work into a comparable framework.
- Gives a clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) for engineering practice.
- Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration.
- Lists open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Datasets commonly used in this area: MS MARCO, BEIR, Natural Questions, domain corpora
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs
- A Survey on Multi-Turn Interaction Capabilities of Large Language Models (arXiv:2501.09959)
- Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey (arXiv:2503.22458)
- An Empirical Analysis on Multi-turn Conversational Recommender Systems (DOI: 10.1145/3626772.3657893)
- Beyond Whole Dialogue Modeling: Contextual Disentanglement for Conversational Search (arXiv:2504.17427)
- CHIQ: Contextual History Enhancement for Improving Query Rewriting (arXiv:2406.05013)
- Few-Shot Conversational Dense Retrieval, SIGIR 2021 (DOI: 10.1145/3404835.3462856)
- Original paper: Aligning Query Representation with Rewritten Query and Relevance Judgments in Conversational Search, CIKM 2024.
Experimental Setup (as outlined in the post)
Note: specific numerical results should be verified against the original PDF; this post is based on the abstract and public metadata.
Insights for Search / Recommendation / Personalization
1. Architecture: cascaded retrieval + rerank + generation remains mainstream, but the agentic paradigm makes "number and strategy of retrieval steps" itself learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data needs safeguards against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Product: latency, cost, interpretability, and safety are hard constraints for industrial deployment — one cannot optimize academic benchmarks alone.
Engineering Checklist (from the post's appendix)
| Check | Question | Suggestion | |-------|----------|------------| | Data | Does training/index data contain PII? How is versioning handled? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascading + early stop, caching hot queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, manual audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU usage per query? | Small-model routing, distillation, hybrid sparse+dense |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Modeling search as sequential decision-making and tool calls | | Gen-IR | Generative Information Retrieval |