Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data

A May 2025 arXiv paper by Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, and Jimmy Lin (University of Waterloo retrieval group).

• Paper: <https://arxiv.org/abs/2505.19274>
• Topic area: Embedding models / dense retrieval

Overview

The paper's central claim is in its title: conventional contrastive learning — the standard training recipe for dense retrieval models — often falls short. The authors propose improving dense retrievers via cross-encoder listwise distillation combined with synthetic data, transferring the stronger relevance judgments of cross-encoders into efficient bi-encoder embeddings.

Background and Motivation

Dense retrieval has become a core component of large-scale search, recommendation, and RAG pipelines. Typical architectures follow a cascade: a bi-encoder retriever provides fast recall via approximate nearest-neighbor search, followed by a cross-encoder reranker for precision. However, bi-encoders trained with standard contrastive objectives can be sensitive to domain shift and long-tail queries, and cannot capture the fine-grained query-document interactions that cross-encoders model. This work addresses that gap by distilling listwise ranking knowledge from a cross-encoder into the retriever, using synthetic data to expand training coverage.

Method Sketch

1. Input and representation: encode queries and documents into dense representations. 2. Core modules: retriever and cross-encoder teacher (reranker). 3. Learning strategy: listwise distillation from cross-encoder scores, contrastive learning, and synthetic data generation. 4. Inference: standard single-pass dense retrieval with ANN search, optionally followed by reranking.

Evaluation Context

Typical evaluation setups in this space include:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora
• Metrics: nDCG@10, MRR, Recall@k, Hit@k
• Baselines: BM25, standard dense retrieval, cross-encoder reranking, commercial search APIs

Exact numerical results should be verified against the original PDF, as this write-up is based on the abstract and public metadata.

Key Takeaways for Search / Rec / Personalization

1. Architecture: cascaded retrieve-rerank-generate remains dominant, but agentic paradigms increasingly treat retrieval strategy itself as learnable. 2. Data: high-quality instruction data and logs matter as much as architecture; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human assessment. 4. Deployment: latency, cost, interpretability, and safety are hard production constraints that academic benchmarks alone do not capture.

Limitations and Open Problems

Potential limitations include experiment scale constrained by GPU budgets, benchmarks that diverge from real user distributions, English-centric data with unknown cross-lingual generalization, and safety risks of agentic systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs, and causal/fairness constraints for recommendation.

Related Entries

• The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual Embedding Models
• A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (arXiv:2402.03216)
• BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search framed as sequential decision-making with tool use | | Gen-IR | Generative Information Retrieval |