Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition (May 2025, arXiv)

This forum entry catalogs and analyzes an academic paper on dense retrieval knowledge acquisition.

Meta Information

| Field | Content | |-------|---------| | Title | Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition | | Authors / Affiliations | Zheng Yao, Shuai Wang, Guido Zuccon | | Source | https://arxiv.org/abs/2505.07166 | | Type | Academic paper | | Category | Embedding models |

Background and Motivation

The paper addresses a long-standing question in information retrieval: whether dense retrievers acquire knowledge during pre-training or during supervised fine-tuning. The study sits at the intersection of large-scale search, recommendation, and personalization systems, where efficiency, scalability, and user-intent understanding remain core challenges. Traditional pipelines split retrieval, ranking, and generation, making it hard to meet LLM-era demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge.

Core Contributions

• A unified perspective that places scattered prior work into a comparable framework.
• A clean decomposition of method components (representation learning, retriever, reranker, planner, generator, feedback mechanisms) to aid engineering adoption.
• Reproducible benchmarks and protocols that lower the entry cost for follow-up research.
• Discussion of interfaces with LLM tool calling, reinforcement learning, and multi-agent collaboration.
• Open problems: evaluation trustworthiness, latency/cost, hallucination and safety, cross-lingual and multimodal extension.

Typical Method Pipeline

The general workflow follows: problem formulation → model/system design → training/construction → inference pipeline.

1. Input and representation: encoding queries, documents, and user context into dense or sparse representations, or structured prompts. 2. Core modules: retrievers, rerankers, planners, memory modules, tool interfaces. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, RL (including process rewards), bootstrap data synthesis. 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping and budget control.

Evaluation

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation sets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
• Ablations: contribution of retrieval steps, reranking depth, and training data scale.

Quantitative results should be verified against the original PDF; this summary is based on the abstract and public metadata.

Key Insights for Search / Rec / Personalization

1. Architecture: cascaded retrieval + rerank + generation remains mainstream, but agentic paradigms treat retrieval frequency and policy itself as learnable. 2. Data: high-quality instruction data and click/session logs matter equally; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment.

Related Entries

• The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual Embedding Models
• A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (arXiv:2402.03216)
• BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking-quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |

References

• Original paper: Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition, arXiv.