French MTEB Repository: A Benchmark Suite for French Text Embeddings

Forum topic · 小凯 · 2026-07-05

Summary

The French MTEB repository, maintained by Lyon-NLP and hosted on GitHub, extends the Massive Text Embedding Benchmark (MTEB) to the French language. It provides a standardized, reproducible evaluation framework for measuring the quality of French text embedding models across tasks relevant to information retrieval, such as semantic search, retrieval-augmented generation (RAG), recommendation, and agent-based systems. The project addresses a gap in embedding evaluation, where most benchmarks are English-centric and cross-lingual generalization remains unverified. The repository supports the typical pipeline of encoding queries and documents into dense representations, retrieving candidates, and reranking, enabling fair comparison of models under consistent protocols. For researchers, it offers reproducible baselines and datasets; for engineers, it helps assess which embedding models are suitable for production French-language search and RAG stacks; for product teams, it provides evidence beyond offline nDCG metrics. The post situates French MTEB alongside related multilingual embedding work such as BGE M3, Arctic-Embed 2.0, and Scandinavian embedding benchmarks, and lists open challenges including evaluation reliability, latency and cost, hallucination and safety, and multilingual/multimodal extension. Repository: https://github.com/Lyon-NLP/mteb-french

French MTEB Repository

Repository: https://github.com/Lyon-NLP/mteb-french
Type: Open-source project
Category: Embedding models
Focus: Information retrieval evaluation for French

Overview

French MTEB is a benchmark repository that adapts the Massive Text Embedding Benchmark (MTEB) methodology to the French language. It provides a unified, reproducible framework for evaluating French text embedding models across tasks such as retrieval, clustering, and semantic similarity, enabling fair comparison of models for French-language search and RAG systems.

Why It Matters

Most embedding benchmarks are English-centric, making cross-lingual generalization of embedding models uncertain. French MTEB addresses this by offering:

A unified evaluation framework that brings scattered French NLP tasks into a comparable benchmark.
Reproducible baselines and datasets, lowering the entry cost for researchers.
Clear interfaces with modern paradigms such as LLM tool use, RAG, and agentic search, where embedding quality directly affects downstream task success.

How It Fits the Retrieval Pipeline

Within a typical pipeline — query/document encoding into dense or sparse representations, candidate retrieval, reranking, and generation — the benchmark evaluates the representation and retrieval stages under consistent protocols (e.g., nDCG for ranking quality).

Related Work

The Scandinavian Embedding Benchmarks (multilingual regional benchmarks)
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (Dec 2024)
BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity
BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners

Open Challenges

Evaluation reliability and benchmark-vs-real-user distribution mismatch
Latency and cost budgets in production serving
Hallucination and safety in open retrieval
Cross-lingual and multimodal extension

Practical Guidance

For engineers adopting embedding models benchmarked here: consider index update frequency, embedding version compatibility, online A/B sensitivity, and failure modes (empty retrieval, over-generation). Treat benchmark results as directional evidence and validate with shadow experiments before full rollout.

Repository: https://github.com/Lyon-NLP/mteb-french