MTEB: The Massive Text Embedding Benchmark (arXiv 2210.07316, Oct 2022) and Its Leaderboard
Metadata
| Field | Content | |------|------| | Title | MTEB: Massive Text Embedding Benchmark | | Authors / Affiliations | Niklas Muennighoff, Nouamane Tazi, Loïc Magne, Nils Reimers | | Published | October 2022 (arXiv) | | Source | https://arxiv.org/abs/2210.07316 | | Resource type | Academic paper | | Section | Embedding models |
Overview
MTEB (Massive Text Embedding Benchmark) addresses a core pain point in text embedding research: evaluation fragmentation. Before MTEB, embedding models were usually validated on a single task (e.g., semantic search via BEIR, or similarity via STS), so it was unclear how well a model strong on one task would generalize to others.
MTEB contributes:
- A unified benchmark spanning many task families: classification, clustering, pair classification, reranking, retrieval, semantic textual similarity (STS), and summarization, with dozens of datasets.
- A public leaderboard (https://huggingface.co/spaces/mteb/leaderboard) ranking models by per-task and mean scores.
- Findings showing that no single model dominates all tasks — the best model for retrieval may underperform on clustering or classification — motivating multi-task evaluation as standard practice.
- MTEB became the de facto standard for comparing text embedding models, cited across model releases (Sentence-BERT lineage, BGE, E5, GTE, Arctic-Embed, and more).
- It lowered the entry cost for practitioners choosing embeddings for search, RAG pipelines, and recommendation systems by providing reproducible protocols.
- It spawned follow-ups, including multilingual MTEB variants and regional benchmarks such as the Scandinavian Embedding Benchmarks.
- Datasets: Per-task public datasets drawn from established sources (e.g., retrieval collections in the style of BEIR/MS MARCO-era benchmarks, STS pairs, clustering corpora).
- Metrics: Task-appropriate metrics such as accuracy/F1 (classification), V-measure (clustering), nDCG@10 / Recall@k (retrieval), Spearman correlation (STS).
- Baselines: Prior sentence embedding models (e.g., SimCSE, Sentence-T5) and cross-encoder / unsupervised baselines.
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual Embeddings
- Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
- BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (arXiv:2402.03216)
- BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
- Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)
- Original paper: MTEB: Massive Text Embedding Benchmark. https://arxiv.org/abs/2210.07316
Why It Matters
Evaluation Design (typical protocol)
Exact numbers should be verified against the paper's tables and the live leaderboard, since model rankings change continuously as new models are submitted.
Takeaways for Practitioners
1. Evaluate on multiple tasks: single-benchmark wins do not guarantee general-purpose embedding quality. 2. Match model choice to workload: retrieval-optimized embeddings may trade off clustering or STS performance. 3. Watch leaderboard churn: rankings shift as new models are added; validate on your own domain data before adoption. 4. Benchmarks are not deployment: latency, cost, multilingual coverage, and index-update workflows remain engineering constraints beyond MTEB scores.