Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data

Forum topic · 小凯 · 2026-07-05

Summary

This forum post introduces a May 2025 arXiv paper (arXiv:2505.19274) by Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, and Jimmy Lin from the University of Waterloo. The paper argues that conventional contrastive learning used to train dense retrieval models often falls short, and proposes improving dense retrievers through knowledge distillation from cross-encoders using listwise objectives combined with synthetic training data. Cross-encoders model fine-grained query-document interactions but are too expensive for first-stage retrieval, so distilling their ranking quality into efficient bi-encoder dense retrievers is a practical strategy. The post situates the work within the broader evolution of neural information retrieval, from BM25 and BERT-style cross-encoders to bi-encoder dense retrieval, late interaction, and LLM-era retrieval-augmented generation. It also provides engineering checklists covering data privacy, latency budgets, quality evaluation, safety, and cost, along with a glossary of IR/RAG terminology and cross-references to related embedding model papers such as BGE-M3 and Arctic-Embed 2.0. Readers are advised to verify quantitative results against the original PDF, as this post primarily summarizes methodology and context from public metadata.

This post summarizes an academic paper entry from an awesome-list of embedding and retrieval research.

Paper Information

Title: Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data (May 2025, arXiv)
Authors / Affiliations: Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Jimmy Lin
Link: https://arxiv.org/abs/2505.19274
Category: Embedding models

Key Points

The paper contends that conventional contrastive learning, the standard training recipe for dense retrieval models, often underperforms.
The proposed remedy is cross-encoder listwise distillation: transferring the superior ranking ability of cross-encoders into efficient bi-encoder dense retrievers using listwise objectives.
Synthetic data supplements the training pipeline to further improve retriever quality.
The work targets the classic IR trade-off: cross-encoders capture fine-grained query-document interactions but cannot pre-compute document representations, while dense retrievers enable fast ANN recall but are sensitive to domain shift and long-tail queries.

Context in Neural IR

The post situates the paper in the evolution of neural retrieval: BM25 → BERT cross-encoders → bi-encoder dense retrieval → late interaction → generative retrieval and LLM agents. Each generation balances efficiency, effectiveness, and maintainability. In the LLM era, additional variables include inference budget and action space (whether to retrieve, how many times, which tools to call).

Engineering Checklist (from the post)

| Area | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Version management? | Partitioned indexes, sanitization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascading + early stopping, hot-query caching, async reranking | | Quality | Do offline gains transfer to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias via open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Related Entries

The Scandinavian Embedding Benchmarks
A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
BGE M3-Embedding (arXiv:2402.03216)
BGE-en-ICL / BGE-ICL few-shot embeddings (arXiv:2409.15700)
Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)

Recommendations for Readers

1. Researchers: reproduce core comparisons; check whether statistical significance and compute cost are reported. 2. Engineers: extract pluggable modules (encoder, reranker, planner) and assess integration cost with existing stacks. 3. Product managers: focus on user-perceivable gains (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.

> Note: The forum post is largely a metadata-driven summary; quantitative results should be verified against the original PDF.

Glossary

IR: Information Retrieval
RAG: Retrieval-Augmented Generation
LTR: Learning to Rank
nDCG: Normalized Discounted Cumulative Gain
Agentic Search: modeling search as sequential decisions and tool calls
Gen-IR: Generative Information Retrieval