C-MTEB: Chinese Massive Text Embedding Benchmark Repository (FlagEmbedding)

Forum topic · 小凯 · 2026-07-05

Summary

C-MTEB (Chinese MTEB) is the Chinese-language extension of the Massive Text Embedding Benchmark, hosted in the FlagOpen/FlagEmbedding repository on GitHub. It provides a standardized suite of Chinese embedding evaluation tasks covering information retrieval, semantic textual similarity, reranking, classification, clustering, and pair classification, enabling reproducible comparison of Chinese text embedding models. C-MTEB is widely used to evaluate models from the BGE (BAAI General Embedding) family, including BGE-M3 and instruction-tuned variants, alongside other open-source Chinese embedding models. The repository includes evaluation code, task definitions, and leaderboards, lowering the entry cost for researchers benchmarking on Chinese corpora. This entry situates C-MTEB within the broader information retrieval ecosystem: dense retrieval, reranking, retrieval-augmented generation (RAG), and agentic search. It also lists related resources such as the Scandinavian Embedding Benchmarks, Arctic-Embed 2.0, and BGE M3-Embedding. Practitioners are advised to treat benchmark results as directional evidence and validate with shadow deployments and A/B testing, accounting for query distribution drift, cold-start items, and multilingual user populations.

C-MTEB: Chinese Massive Text Embedding Benchmark Repository

Overview

C-MTEB (Chinese MTEB) is the Chinese-language benchmark suite maintained under the FlagEmbedding project. It adapts the MTEB (Massive Text Embedding Benchmark) methodology to Chinese, providing standardized tasks and evaluation code for text embedding models.

Repository: https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB
Resource type: Open-source project
Category: Embedding models / Information retrieval

Purpose and Motivation

Large-scale search, recommendation, and personalization systems face long-standing challenges in efficiency, scalability, and user intent understanding. Traditional pipelines treat retrieval, ranking, and generation as separate stages, which struggles to meet the LLM-era demand for natural-language interaction, multi-hop reasoning, and real-time knowledge. C-MTEB addresses this gap by offering a unified, reproducible benchmark for Chinese embedding quality.

Core Contributions

A unified framework consolidating Chinese embedding tasks into a comparable benchmark
Decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms
Reproducible baselines and task suites that lower the entry cost for subsequent research
Interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration
Explicit open problems: evaluation reliability, latency and cost, hallucination and safety, and cross-lingual/multilingual extension

Typical System Pipeline

1. Input and representation: encode queries, documents, and user context into dense or sparse representations, or structured prompts 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces combined serially or in parallel 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, RL (including process rewards), and bootstrapped data synthesis 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control

Installation and API

See the project README and documentation at the repository link above.

Related Entries

The Scandinavian Embedding Benchmarks
A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (arXiv:2402.03216)
BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)

Engineering Checklist

| Concern | Question | Recommendation | |---------|----------|----------------| | Data | PII in training/index data? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval depth? | Cascades + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to CTR/satisfaction? | Interleaving experiments, manual audits, citation checks | | Safety | Open retrieval introducing poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Glossary

IR: Information Retrieval
RAG: Retrieval-Augmented Generation
LTR: Learning to Rank
nDCG: Normalized Discounted Cumulative Gain
Agentic Search: search modeled as sequential decision-making with tool calls
Gen-IR: Generative Information Retrieval

Recommendations

1. Researchers: reproduce the core comparisons and check for statistical significance and compute-cost reporting 2. Engineers: evaluate pluggable modules (encoders, rerankers, planners) and integration cost with existing stacks 3. Product managers: focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone

> Note: Benchmark results should be treated as directional evidence. Production migration requires recalibration for query distribution drift, seasonality, cold-start items, and multilingual user mixes — validate with shadow experiments and A/B tests before full rollout.