REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Models (arXiv, Oct 2024)

• Paper: https://arxiv.org/abs/2410.12890
• Authors: Ambuje Gupta, Mrinal Rawat, Andreas Stolcke, Roberto Pieraccini
• Category: Embedding models

Summary

REFINE (Retrieval Enhancement through Fine-Tuning via model fusIoN of Embedding models) targets the problem that fine-tuning text embedding models for retrieval usually demands large volumes of labeled training data, which domain-specific applications rarely have. The proposed approach fine-tunes embedding models on small amounts of data and applies model fusion techniques to combine the resulting models, improving retrieval performance on scarce data.

Key points

• Problem: Embedding models pretrained on general corpora underperform on domain-specific retrieval, but supervised fine-tuning requires large labeled query–passage datasets that are expensive to collect.
• Approach: Instead of a single fine-tuned model, REFINE leverages model fusion — combining multiple fine-tuned embedding model variants — to obtain a stronger retriever from limited training data.
• Evaluation: The paper reports retrieval experiments showing that fused models fine-tuned on scarce data outperform single fine-tuned models; exact metrics and dataset details should be checked in the original PDF.

Context within the embedding literature

This work fits into a broader line of research on adapting embedding models for retrieval, including:

• BGE M3-Embedding: multi-lingual, multi-functionality, multi-granularity embeddings (arXiv:2402.03216)
• Arctic-Embed 2.0: multilingual retrieval (arXiv:2412.04506)
• BGE-en-ICL / BGE-ICL: making text embedders few-shot learners (arXiv:2409.15700)
• The Scandinavian Embedding Benchmarks: multilingual evaluation of embedding models
• Beyond Benchmarks: evaluating embedding model similarity for retrieval (arXiv:2407.08275)

Notes for practitioners

• Model fusion offers a data-efficient alternative when annotation budgets are small; consider it when a single fine-tuning run plateaus.
• Validate fused models against strong baselines (BM25, off-the-shelf dense retrievers) on your own domain data before adoption.
• As with any embedding swap, plan for re-indexing and versioning of your vector store.

Reference

• Gupta, A., Rawat, M., Stolcke, A., Pieraccini, R. *REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Models*. arXiv:2410.12890, October 2024. https://arxiv.org/abs/2410.12890