M3-Embedding: Multi-Lingual, Multi-Functional, Multi-Granularity Text Embeddings via Self-Knowledge Distillation
Overview
- Paper: M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Authors: Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu
- Published: 2024-02-05
- Source: https://arxiv.org/abs/2402.03216
- Category: Embedding models
- Self-knowledge distillation: Relevance scores produced by the different retrieval functionalities are integrated as a teacher signal, improving training quality beyond any single retrieval mode.
- Optimized batching strategy: Enables large batch sizes and high training throughput, which improves the discriminativeness of the learned embeddings.
- A single model covering dense + sparse + multi-vector retrieval simplifies hybrid search stacks and removes the need to maintain separate retrievers.
- 8,192-token context support makes it suitable for long-document RAG indexing.
- Broad language coverage reduces the engineering cost of supporting cross-lingual enterprise search.
- Self-knowledge distillation is a practical recipe: combine the strengths of multiple retrieval modes into one embedding space.
- The Scandinavian Embedding Benchmarks
- Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
- BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners
- Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval
- Original paper: arXiv:2402.03216
Key Capabilities
M3-Embedding is defined by three "multi" properties:
1. Multi-Linguality: Uniform support for semantic retrieval across 100+ working languages. 2. Multi-Functionality: Simultaneously supports the three common retrieval modes — dense retrieval, multi-vector retrieval, and sparse retrieval — within a single model. 3. Multi-Granularity: Handles inputs ranging from short sentences to long documents of up to 8,192 tokens.
Technical Contributions
Results
The model achieves new state-of-the-art results on multilingual, cross-lingual, and long-document retrieval benchmarks.
Original Abstract (verbatim)
> In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in *Multi-Linguality*, *Multi-Functionality*, and *Multi-Granularity*. It provides a uniform support for the semantic retrieval of more than 100 working languages. It can simultaneously accomplish the three common retrieval functionalities: dense retrieval, multi-vector retrieval, and sparse retrieval. Besides, it is also capable of processing inputs of different granularities, spanning from short sentences to long documents of up to 8,192 tokens. The effective training of M3-Embedding presents a series of technical contributions. Notably, we propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, which enables a large batch size and high training throughput to improve the discriminativeness of embeddings. M3-Embedding exhibits a superior performance in our experiment, leading to new state-of-the-art results on multilingual, cross-lingual, and long-document retrieval benchmarks.