Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation (ConvAug)

Overview

• Paper: Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation
• Authors: Haonan Chen, Zhicheng Dou, Kelong Mao, Jiongnan Liu, Ziliang Zhao
• Published: 2024-02-11
• Link: https://arxiv.org/abs/2402.07092
• Code: https://github.com/haon-chen/ConvAug
• Topic: Conversational Search

Abstract (original)

> Conversational search utilizes muli-turn natural language contexts to retrieve relevant passages. Existing conversational dense retrieval models mostly view a conversation as a fixed sequence of questions and responses, overlooking the severe data sparsity problem -- that is, users can perform a conversation in various ways, and these alternate conversations are unrecorded. Consequently, they often struggle to generalize to diverse conversations in real-world scenarios. In this work, we propose a framework for generalizing Conversational dense retrieval via LLM-cognition data Augmentation (ConvAug). ConvAug first generates multi-level augmented conversations to capture the diverse nature of conversational contexts. Inspired by human cognition, we devise a cognition-aware process to mitigate the generation of false positives, false negatives, and hallucinations. Moreover, we develop a difficulty-adaptive sample filter that selects challenging samples for complex conversations, thereby giving the model a larger learning space. A contrastive learning objective is then employed to train a better conversational context encoder. Extensive experiments conducted on four public datasets, under both normal and zero-shot settings, demonstrate the effectiveness, generalizability, and applicability of ConvAug.

Key Points

• Problem: Conversational dense retrieval models treat a conversation as a fixed sequence of questions and responses, suffering from data sparsity — alternative conversation paths users could take are unrecorded — leading to poor generalization in real-world scenarios.
• Proposed solution (ConvAug): An LLM-based data augmentation framework that generates multi-level augmented conversations to reflect the diversity of conversational contexts.
• Cognition-aware generation: Inspired by human cognition, a dedicated process reduces false positives, false negatives, and hallucinations in the synthetic data.
• Difficulty-adaptive sample filter: Selects challenging samples for complex conversations, expanding the model's learning space.
• Training objective: A contrastive learning objective is used to train a better conversational context encoder.
• Evaluation: Extensive experiments on four public datasets under both normal and zero-shot settings show effectiveness, generalizability, and applicability.

Context and Significance

This work sits at the intersection of conversational search and LLM-era retrieval systems. Rather than splitting retrieval, ranking, and generation into rigid pipelines, it focuses on the data side: using LLMs as synthetic conversation generators while controlling generation quality through cognition-inspired safeguards. Related entries include surveys on conversational search, ChatRetriever, and agentic conversational search approaches.

Takeaways for Practitioners

1. Researchers: Reproduce the core baselines; check whether statistical significance and compute costs are reported. 2. Engineers: The augmentation pipeline (generation → filtering → contrastive training) can be adapted to improve context encoders in existing stacks. 3. Product teams: Focus on user-perceivable benefits such as multi-turn consistency and answer trustworthiness, not just offline nDCG.

Limitations to Note

Synthetic data generation requires LLM compute and must guard against knowledge leakage and distribution shift; zero-shot generalization is tested on four public datasets, but results may not fully match real user distributions.

References

• Original paper: Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation. https://arxiv.org/abs/2402.07092