Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation (ConvAug)

Authors: Haonan Chen, Zhicheng Dou, Kelong Mao, Jiongnan Liu, Ziliang Zhao Published: 2024-02-11 Paper: https://arxiv.org/abs/2402.07092 Code: https://github.com/haon-chen/ConvAug

Overview

Conversational search uses multi-turn natural language context to retrieve relevant passages. Existing conversational dense retrieval models treat a conversation as a fixed sequence of questions and responses, ignoring the severe data sparsity problem: users can hold the same conversation in many different ways, and these alternative conversations are unrecorded in training data. As a result, such models often fail to generalize to diverse conversations in real-world scenarios.

Proposed Method: ConvAug

The authors propose ConvAug, a framework for generalizing conversational dense retrieval via LLM-cognition data augmentation. Its main components are:

1. Multi-level conversation augmentation — an LLM generates multi-level augmented conversations to capture the diverse nature of conversational contexts. 2. Cognition-aware generation process — inspired by human cognition, this process mitigates the generation of false positives, false negatives, and hallucinations during augmentation. 3. Difficulty-adaptive sample filter — selects challenging samples for complex conversations, giving the model a larger learning space. 4. Contrastive learning objective — used to train a better conversational context encoder.

Experiments

Extensive experiments on four public datasets, under both normal and zero-shot settings, demonstrate the effectiveness, generalizability, and applicability of ConvAug.

Significance

ConvAug shows that LLM-driven data augmentation, when guarded by cognition-aware filtering, can substantially alleviate conversational data sparsity — a promising direction for improving robustness of retrieval models without collecting new human conversation logs.

Original Abstract

> Conversational search utilizes muli-turn natural language contexts to retrieve relevant passages. Existing conversational dense retrieval models mostly view a conversation as a fixed sequence of questions and responses, overlooking the severe data sparsity problem -- that is, users can perform a conversation in various ways, and these alternate conversations are unrecorded. Consequently, they often struggle to generalize to diverse conversations in real-world scenarios. In this work, we propose a framework for generalizing Conversational dense retrieval via LLM-cognition data Augmentation (ConvAug). ConvAug first generates multi-level augmented conversations to capture the diverse nature of conversational contexts. Inspired by human cognition, we devise a cognition-aware process to mitigate the generation of false positives, false negatives, and hallucinations. Moreover, we develop a difficulty-adaptive sample filter that selects challenging samples for complex conversations, thereby giving the model a larger learning space. A contrastive learning objective is then employed to train a better conversational context encoder. Extensive experiments conducted on four public datasets, under both normal and zero-shot settings, demonstrate the effectiveness, generalizability, and applicability of ConvAug.