French MTEB: A French Benchmark Repository for Embedding Models
This entry catalogs the French MTEB repository, an open-source project for evaluating text embedding models on French-language tasks.
Metadata
| Field | Value | |-------|-------| | Project | French MTEB repository | | Repository | https://github.com/Lyon-NLP/mteb-french | | Type | Open-source project | | Category | Embedding models | | Focus | Information retrieval, embedding evaluation |
What it is
French MTEB brings the Massive Text Embedding Benchmark (MTEB) methodology to French, addressing the English-centric bias that still dominates embedding model evaluation. It is useful for anyone building French-language search, recommendation, RAG, or agent systems that depend on high-quality sentence and document embeddings.
- Repository: https://github.com/Lyon-NLP/mteb-french
- Installation, API, and comparisons: see the project README and documentation
- The Scandinavian Embedding Benchmarks — comprehensive multilingual assessment
- Arctic-Embed 2.0 — multilingual retrieval without compromise (arXiv:2412.04506)
- BGE M3-Embedding — multi-lingual, multi-functionality, multi-granularity (arXiv:2402.03216)
- BGE-en-ICL / BGE-ICL — making text embedders few-shot learners (arXiv:2409.15700)
- Beyond Benchmarks — evaluating embedding model similarity for retrieval (arXiv:2407.08275)
- Offline metric gains (e.g., nDCG) do not always translate to online CTR or user satisfaction—validate with interleaving experiments and manual audits.
- Benchmark distributions often differ from real user queries; watch for distribution drift and long-tail queries.
- English-centric training data leaves cross-lingual generalization largely unknown.
- Consider latency budgets (cascaded retrieval, caching, early stopping) and cost (per-query tokens, GPU usage) before full rollout.
- Project page: https://github.com/Lyon-NLP/mteb-french
Context in the embedding ecosystem
The entry cross-references related multilingual and domain-specific embedding work:
Why it matters
Most embedding benchmarks are built around English data, making cross-lingual generalization claims unreliable. Language-specific benchmarks like French MTEB provide reproducible task suites so that models can be compared fairly on French retrieval, clustering, classification, and related tasks—reducing the entry cost for follow-up research and helping practitioners pick models for production French-language systems.
Practical considerations (from the source discussion)
When evaluating embedding benchmarks and applying them in production, keep in mind:
Related glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decisions and tool calls | | Gen-IR | Generative Information Retrieval |