C-MTEB: Chinese Massive Text Embedding Benchmark Repository (FlagEmbedding)

Forum topic · 小凯 · 2026-07-05

Summary

C-MTEB (Chinese MTEB) is the Chinese-language extension of the Massive Text Embedding Benchmark, hosted within the open-source FlagEmbedding repository by BAAI at github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB. The benchmark provides a unified evaluation suite for Chinese text embedding models across tasks such as information retrieval, semantic similarity, classification, clustering, reranking, and sentence-pair tasks, enabling reproducible comparison of embedding models on Chinese data. This forum entry indexes the repository under the 'Embedding models' section of a curated awesome list, alongside related resources such as BGE M3-Embedding, Arctic-Embed 2.0, and Scandinavian embedding benchmarks. The accompanying discussion situates C-MTEB in the broader evolution of neural information retrieval — from BM25 and dense dual-tower retrieval to LLM-era RAG and agentic search — and offers practical engineering guidance on latency budgets, index versioning, security filtering, and cost control when deploying embedding-based systems. It emphasizes treating benchmark results as directional evidence to be validated through shadow experiments and A/B testing before production rollout.

C-MTEB: Chinese Massive Text Embedding Benchmark Repository

C-MTEB is the Chinese-language counterpart of MTEB (Massive Text Embedding Benchmark), maintained inside the open-source FlagEmbedding repository.

Overview

| Field | Content | |---|---| | Title | Chinese MTEB (C-MTEB) repository | | Repository | https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB | | Resource type | Open-source project | | Section | Embedding models | | Use cases | Search, recommendation, RAG, agents |

What C-MTEB provides

A unified evaluation framework for Chinese text embedding models, bringing scattered work into a comparable benchmark suite.
Coverage of core embedding task families relevant to information retrieval: retrieval, semantic similarity, reranking, classification, and clustering.
Reproducible benchmark protocols and datasets that lower the entry cost for researchers and engineers evaluating Chinese embedding models.

Context: where C-MTEB sits

Neural information retrieval has evolved from BM25 through BERT cross-encoders and dense dual-tower retrieval to late-interaction, generative retrieval, and LLM-based agentic search. Each generation balances the efficiency–effectiveness–maintainability triangle. Benchmarks like C-MTEB anchor this progress by providing static, reproducible evaluation, complementing newer process-oriented metrics (task success rate, citation accuracy, multi-hop reasoning) used in RAG and agentic search settings.

Engineering checklist for deploying embedding systems

| Check | Question | Suggestion | |---|---|---| | Data | Does training/indexed data contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascaded retrieval with early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Security | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Related entries

The Scandinavian Embedding Benchmarks
A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
BGE M3-Embedding (arXiv:2402.03216)
BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)

Reader guidance

1. Researchers: reproduce the benchmark comparisons and check whether statistical significance and compute costs are reported. 2. Engineers: treat components (encoders, rerankers, planners) as pluggable modules and evaluate integration cost with existing stacks. 3. Product managers: focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.

Benchmark results should be treated as directional evidence: validate via small-scale shadow experiments and A/B tests before full production rollout, since query distribution drift, seasonality, cold-start items, and multilingual user mixes can shift optimal strategies.

Glossary

| Term | Meaning | |---|---| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |