MMTEB: Massive Multilingual Text Embedding Benchmark on Hugging Face (Feb 2025)
MMTEB (Massive Multilingual Text Embedding Benchmark) was released in February 2025 on Hugging Face as a large-scale expansion of the popular MTEB embedding evaluation suite, accompanied by public leaderboards.
Key facts
- 1,043 languages covered in total
- 550 tasks, including:
- Bitext mining (text pairing) — the primary category
- 255 classification tasks
- 209 clustering tasks
- 142 retrieval tasks
- Task variety ranges from sentiment analysis and question-answering reranking to long-document retrieval
- 17 domains, including legal, religious, programming, web, social, medical, blog, and academic content
- MTEB(eng, v2) — a new English benchmark that is much smaller and faster than the original English MTEB, making submissions significantly cheaper and simpler
- MTEB(Multilingual, v1)
- MTEB(Law, v1)
- Paper: https://huggingface.co/papers/2502.13595
Sub-benchmarks
The task collection is subdivided into specialized benchmarks, each with its own leaderboard:
Source and links
Announced by Tom Aarsen (Hugging Face) via LinkedIn. Paper page and leaderboard:
Why it matters
MMTEB standardizes embedding model evaluation across languages, task types, and domains. Its breadth makes it a reference point for anyone benchmarking sentence embeddings for search, retrieval-augmented generation (RAG), semantic similarity, and multilingual NLP applications. The streamlined MTEB(eng, v2) in particular lowers the cost barrier for community submissions and frequent model evaluation.