ConvAug: Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation
Overview
- Paper: Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation
- Authors: Haonan Chen, Zhicheng Dou, Kelong Mao, Jiongnan Liu, Ziliang Zhao
- Published: 2024-02-11
- arXiv: https://arxiv.org/abs/2402.07092
- Code: https://github.com/haon-chen/ConvAug
- Topic: Conversational Search / Dense Retrieval
- Data augmentation with LLMs is an effective route to break the fixed-sequence assumption in conversational retrieval training data.
- The cognition-aware filtering shows that naive LLM augmentation risks label noise (false positives/negatives) and hallucinations, which must be actively controlled.
- Difficulty-aware sample selection plus contrastive learning improves encoder robustness, including in zero-shot transfer to unseen datasets.
- Paper: <https://arxiv.org/abs/2402.07092>
- Code: <https://github.com/haon-chen/ConvAug>
Motivation
Conversational search retrieves relevant passages using multi-turn natural language context. Existing conversational dense retrieval models mostly view a conversation as a fixed sequence of questions and responses, overlooking a severe data sparsity problem: users can conduct the same conversation in many different ways, and these alternative conversation paths are unrecorded in training data. As a result, such models often fail to generalize to the diverse conversations seen in real-world scenarios.
Method: ConvAug
ConvAug (Conversational dense retrieval via LLM-cognition data Augmentation) tackles this with three main components:
1. Multi-level conversation augmentation — An LLM generates multiple levels of augmented conversations to capture the diverse nature of conversational contexts. 2. Cognition-aware generation process — Inspired by human cognition, this process mitigates the generation of false positives, false negatives, and hallucinations during augmentation. 3. Difficulty-adaptive sample filter — Selects challenging samples for complex conversations, giving the model a larger learning space.
A contrastive learning objective is then used to train a better conversational context encoder.
Results
Extensive experiments on four public datasets, under both normal and zero-shot settings, demonstrate the effectiveness, generalizability, and applicability of ConvAug compared to existing conversational dense retrieval baselines.