KaLM-Embedding-V2: Superior Training Techniques and Data Inspire a Versatile Embedding Model
Source: arXiv:2506.20923 — June 2025
Overview
KaLM-Embedding-V2 is a versatile text embedding model introduced in June 2025. The paper's central claim is that superior training techniques and higher-quality data—rather than sheer model scale—can produce a general-purpose embedding model that performs strongly across retrieval and semantic similarity tasks.
Key points
- Compact backbone: The model is built on a small open-source LLM (Qwen2.5-0.5B class), targeting cost-efficient deployment.
- Holistic training recipe: The authors describe a multi-stage pipeline including data cleaning and deduplication, offline pretraining oriented toward semantic textual similarity (STS), and multi-task fine-tuning covering retrieval and related embedding tasks.
- On-policy learning: A further training stage refines the model beyond standard supervised fine-tuning.
- Strong benchmark results: The paper reports state-of-the-art performance among models of comparable size on Chinese embedding benchmarks (MTEB / CMTEB), spanning retrieval, reranking, clustering, and classification.
- Positioning: The work sits alongside other multilingual and Chinese embedding efforts such as BGE-M3, Arctic-Embed 2.0, and BGE-ICL, with a focus on maximizing quality per parameter.
- BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity
- Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
- BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners
- The Scandinavian Embedding Benchmarks
Why it matters
For practitioners building search, RAG pipelines, or recommendation systems, KaLM-Embedding-V2 demonstrates that sub-billion-parameter embedding models can remain competitive, reducing inference latency and cost. For researchers, it underscores that data curation, staged training, and on-policy refinement are levers at least as important as backbone scale.