English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

In-Depth Comparison of Small On-Device Embedding Models (2025–2026)

Forum topic · ✨步子哥 · 2026-07-12

Summary

This study benchmarks 17 compact embedding models suitable for on-device deployment in retrieval-augmented generation (RAG), addressing privacy, cost, and offline constraints of cloud APIs. It defines four on-device thresholds: model size, vector dimension, context length, and hardware/quantization support. Models are evaluated on MTEB, MMTEB, and C-MTEB benchmarks and grouped into ultra-light (<100MB), lightweight multilingual (100–400MB), and near-production (400MB–1.2GB) tiers. Highlights include all-MiniLM-L6-v2, gte-small, bge-small-en/zh-v1.5, multilingual-e5-small, Snowflake Arctic-embed xs/s, jina-embeddings-v2-small-en, mxbai-embed-xsmall-v1, nomic-embed-text-v1.5, bge-m3, EmbeddingGemma-308M, gte-multilingual-base, granite-embedding-278m, nomic-embed-text-v2-moe, mxbai-embed-large-v1, and Qwen3-Embedding-0.6B. The report covers deployment via ONNX, GGUF/llama.cpp, and transformers.js v4, and provides a decision tree mapping use cases to recommended models, with Qwen3-Embedding-0.6B (C-MTEB 71.02, 32K context) named the current overall best.

Key Points

  • On-device definition: An embedding model is "on-device ready" only when it meets four thresholds: disk size, vector dimension, context length, and CPU/NPU viability with quantization.
  • Total landscape: 17 representative models are compared, including all-MiniLM-L6-v2, gte-small, bge-small-en-v1.5, bge-small-zh-v1.5, multilingual-e5-small, snowflake-arctic-embed-xs/s, jina-embeddings-v2-small-en, mxbai-embed-xsmall-v1, nomic-embed-text-v1.5, bge-m3, EmbeddingGemma-308M, gte-multilingual-base, granite-embedding-278m, nomic-embed-text-v2-moe, mxbai-embed-large-v1, and Qwen3-Embedding-0.6B.
  • Ultra-light tier (<100MB): Suitable for browsers, Raspberry Pi, and CI pipelines. gte-small (~70MB) is the best value general English model; bge-small-zh-v1.5 (139MB) is recommended for Chinese; multilingual-e5-small (140MB) covers 100+ languages; jina-embeddings-v2-small-en offers 8K context at 33M parameters; mxbai-embed-xsmall-v1 is optimized for transformers.js WebGPU inference.
  • Lightweight multilingual tier (100–400MB): The sweet spot for personal RAG. nomic-embed-text-v1.5 (274MB, 8K context, MRL down to 64–768 dims) is the recommended default for English on CPU. bge-m3 (1.2GB, 100+ languages) outputs dense, sparse, and ColBERT multi-vectors in one pass, leading C-MTEB at 64.8. EmbeddingGemma-308M (Gemma license, 100+ languages) fits under 200MB RAM after quantization and runs <22ms on EdgeTPU, making it ideal for mobile RAG.
  • Near-production tier (400MB–1.2GB): mxbai-embed-large-v1 (670MB) tops English retrieval MTEB (64.68) for sub-500M models. Qwen3-Embedding-0.6B (MRL 32–1024 dims, 32K context, Apache-2.0, 100+ languages) is the standout: MTEB English v2 70.7, multilingual 64.64, C-MTEB 71.02, with Q4 GGUF at ~400MB. In reported tests it achieved NDCG@10 0.782 on long Chinese documents (vs bge-m3 0.715, nomic-v1.5 0.689) and 0.826 on cross-lingual technical docs.
  • Deployment stack: int8 dynamic quantization cuts size ~75% with 1–3% accuracy loss (1.4–2.1× CPU speedup). ONNX Runtime supports CPU, CUDA, TensorRT, NNAPI (Android NPU), and CoreML (iOS ANE). GGUF via llama.cpp runs Qwen3-0.6B with one-liner llama-embedding. transformers.js v4 with WebGPU delivers ~6ms per query for BGE-small in browser, supported on Chrome/Edge 113+, Firefox (WASM), and Safari 18+ (iOS Safari 26+ for WebGPU).
  • Decision tree summary:
  • Browser/Raspberry Pi/CI: gte-small, all-MiniLM, bge-small-zh-v1.5, or multilingual-e5-small.
  • Personal RAG on laptop CPU: nomic-embed-text-v1.5 for English; Qwen3-Embedding-0.6B (GGUF) or bge-m3 for Chinese/multilingual.
  • Mobile/offline privacy: EmbeddingGemma-308M.
  • English retrieval accuracy on short documents: mxbai-embed-large-v1.
  • Long Chinese docs / cross-lingual / code retrieval: Qwen3-Embedding-0.6B (current overall best).
  • Trends: (1) Small models no longer "settle" — Qwen3-0.6B and EmbeddingGemma lead despite low parameter counts. (2) Three enabling techniques — MRL (truncatable dimensions), long context (8K–32K), and int8/int4 quantization — make small + capable + fast simultaneously. (3) Hybrid retrieval (dense + sparse + multi-vector) is becoming standard. (4) The on-device ecosystem (transformers.js v4 WebGPU, ONNX, GGUF, TFLite/LiteRT/MLX) is mature enough for production privacy-first apps.

Methodology Notes

Scores combine MTEB/MMTEB/C-MTEB official leaderboards, HuggingFace model cards, Ollama observations, and Milvus / MorphLLM / Google Gemma documentation. Version differences (v1/v2) and scope (overall vs retrieval vs multilingual) limit direct comparability; users should re-evaluate on their own data. Document compiled 2026-07-12.

Tags

#embedding-models#rag#on-device-ai#mteb#quantization#transformers-js#onnx-runtime#llama-cpp

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178379402