Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation (ConvAug)
- Authors: Haonan Chen, Zhicheng Dou, Kelong Mao, Jiongnan Liu, Ziliang Zhao
- Published: 2024-02-11
- Paper: arXiv:2402.07092
- Code: github.com/haon-chen/ConvAug
- Data sparsity is a core bottleneck for multi-turn retrieval; LLM-based augmentation offers a scalable way to diversify training conversations.
- The cognition-aware filtering shows that naive LLM data synthesis is insufficient — quality control against hallucinations and labeling errors is essential.
- The difficulty-adaptive curriculum plus contrastive training is a transferable recipe for other retrieval and dialogue modeling tasks.
Overview
Conversational search uses multi-turn natural language context to retrieve relevant passages. Most existing conversational dense retrieval models treat a conversation as a fixed sequence of questions and responses, overlooking a severe data sparsity problem: users can hold a conversation in many different ways, and these alternative conversation paths go unrecorded in training data. As a result, trained models often fail to generalize to the diverse conversations seen in real-world scenarios.
The ConvAug Framework
ConvAug (Conversational dense retrieval via LLM-cognition data Augmentation) tackles this with LLM-driven data augmentation:
1. Multi-level augmented conversations — an LLM generates diverse alternative versions of conversations to capture the varied nature of real conversational contexts. 2. Cognition-aware generation process — inspired by human cognition, this process reduces false positives, false negatives, and hallucinations in the synthetic data. 3. Difficulty-adaptive sample filter — selects challenging samples from complex conversations, giving the model a larger learning space. 4. Contrastive learning objective — trains a better conversational context encoder on the augmented data.
Results
Extensive experiments on four public datasets, under both standard and zero-shot settings, demonstrate ConvAug's effectiveness, generalizability, and applicability compared with existing conversational dense retrieval approaches.
Original Abstract
> Conversational search utilizes muli-turn natural language contexts to retrieve relevant passages. Existing conversational dense retrieval models mostly view a conversation as a fixed sequence of questions and responses, overlooking the severe data sparsity problem -- that is, users can perform a conversation in various ways, and these alternate conversations are unrecorded. Consequently, they often struggle to generalize to diverse conversations in real-world scenarios. In this work, we propose a framework for generalizing Conversational dense retrieval via LLM-cognition data Augmentation (ConvAug). ConvAug first generates multi-level augmented conversations to capture the diverse nature of conversational contexts. Inspired by human cognition, we devise a cognition-aware process to mitigate the generation of false positives, false negatives, and hallucinations. Moreover, we develop a difficulty-adaptive sample filter that selects challenging samples for complex conversations, thereby giving the model a larger learning space. A contrastive learning objective is then employed to train a better conversational context encoder. Extensive experiments conducted on four public datasets, under both normal and zero-shot settings, demonstrate the effectiveness, generalizability, and applicability of ConvAug. The code is released at https://github.com/haon-chen/ConvAug.