Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition (arXiv 2505.07166)

Paper Metadata

| Field | Content | |-------|---------| | Title | Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition | | Authors | Zheng Yao, Shuai Wang, Guido Zuccon | | Published | May 2025 | | Link | https://arxiv.org/abs/2505.07166 | | Type | Academic paper (reproducibility study) | | Section | Embedding models |

One-line Summary

A reproducibility study investigating how dense retrieval models acquire knowledge during pre-training versus fine-tuning, examining which training stage is responsible for memorizing and exploiting document content for retrieval.

Context and Motivation

The paper sits at the intersection of information retrieval and large-scale search/recommendation systems. Dense retrieval pipelines encode queries and documents into vector representations for efficient recall via approximate nearest neighbor search, but they remain sensitive to domain shift and long-tail queries. This reproducibility study systematically examines the respective roles of pre-training and fine-tuning in knowledge acquisition for dense retrievers.

Positioning in the Literature

Neural information retrieval has evolved through several generations, each balancing efficiency, effectiveness, and maintainability:

• Sparse retrieval: BM25 as the classical lexical baseline
• Dual-tower dense retrieval: pre-computed document embeddings with ANN search for millisecond-level recall
• Cross-encoder rerankers: higher accuracy but no pre-computable document representations
• Late interaction and generative retrieval: reducing cascade errors but facing index-update challenges
• RAG and agentic search: extending external knowledge access into an iterable, verifiable, plannable process

The original abstract is quoted verbatim as:

> Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition

Typical Evaluation Setup for Work in This Area

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, plus latency and token cost
• Baselines: BM25, dense retrievers, cross-encoder rerankers, retrieval-free LLMs

*Note: specific numerical results should be verified against the original PDF; this post is based on the abstract and public metadata.*

Engineering Checklist for Dense Retrieval Deployment

| Item | Question | Recommendation | |------|----------|----------------| | Data | Does training/indexing data contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascaded retrieval + early stopping, query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source allowlists, adversarial detection, output filtering | | Cost | Per-query token and GPU footprint? | Small-model routing, distillation, hybrid sparse-dense retrieval |

Related Entries

• The Scandinavian Embedding Benchmarks
• A Universal Framework for Compressing Embeddings in CTR Prediction
• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
• BGE M3-Embedding
• BGE-en-ICL / BGE-ICL Few-Shot Embedding Models
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval

References

• Original paper: Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition. arXiv:2505.07166

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Framing search as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |