MTEB: Massive Text Embedding Benchmark (arXiv Oct 2022, Leaderboard)
Overview
- Paper: MTEB: Massive Text Embedding Benchmark
- Authors: Niklas Muennighoff, Nouamane Tazi, Loic Magne, Nils Reimers
- arXiv: October 2022 — https://arxiv.org/abs/2210.07316
- Type: Academic paper + public leaderboard
- Category: Embedding models
- A unified benchmark spanning dozens of datasets across multiple embedding task families.
- A public leaderboard for transparent, apples-to-apples model comparison.
- Open-source evaluation code so results are reproducible and extendable by the community.
- Empirical insight that no single model dominates all task types, encouraging multi-task evaluation and training.
- Datasets: a wide collection of public benchmarks spanning the six task families above.
- Metrics: task-appropriate metrics such as accuracy, F1, nDCG@10, Recall@k, MAP, and Spearman correlation.
- Baselines: strong open-source sentence embedding models available at the time of release.
- Rankings reflect benchmark performance, which may diverge from real-world user satisfaction on proprietary corpora.
- Benchmark composition and language coverage evolve over time; check the current leaderboard state.
- Production constraints (p99 latency, index size, embedding versioning, safety) require evaluation beyond benchmark scores alone.
- BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (arXiv:2402.03216)
- Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
- BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual Embeddings
- Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)
- A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
- Original paper: https://arxiv.org/abs/2210.07316
One-line summary
MTEB is a massive benchmark for text embeddings covering many task types (classification, clustering, retrieval, reranking, STS, summarization), paired with a public leaderboard for fair, reproducible comparison of embedding models.
Motivation
Before MTEB, text embedding models were typically validated on narrow benchmarks — often only semantic textual similarity or a single retrieval dataset. This made it hard to tell whether a model strong on one task would generalize to clustering, classification, or reranking. MTEB addresses this by evaluating embeddings across a broad, standardized suite of tasks and datasets, so practitioners can pick models based on the capabilities that matter for their applications (search, recommendation, RAG, personalization).
Key contributions
How it works
1. Each dataset defines a task family (classification, clustering, reranking, retrieval, STS, summarization). 2. An embedding model encodes inputs (queries, documents, sentences, etc.) into vectors. 3. Task-specific evaluation protocols score the embeddings (e.g., accuracy for classification, nDCG/recall for retrieval, Spearman correlation for STS). 4. Per-task scores are aggregated to produce overall and per-category rankings on the leaderboard.
Evaluation details
Exact numerical results should be checked in the original paper tables and the live leaderboard, since rankings change as new models are submitted.
Impact and relevance
MTEB has become the reference standard for evaluating embedding models used in large-scale search, recommendation, and RAG pipelines. It spawned numerous follow-ups, including multilingual variants (e.g., MIRACL-style evaluations, BGE M3, Arctic-Embed 2.0) and domain-specific benchmark suites. When selecting an embedding model for production retrieval systems, MTEB scores — alongside latency, cost, and language coverage — are among the first signals most teams consult.