Deep Comparison of Small On-Device Embedding Models (2025-2026)

Forum topic · ✨步子哥 · 2026-07-12

Summary

A comprehensive comparison of 15+ small embedding models suitable for on-device deployment, compiled from MTEB/MMTEB/C-MTEB benchmarks, model cards, and community tests as of July 2026. The guide evaluates models across size, dimensions, context length, multilingual capability, and license, organized into three tiers: ultra-light (<100MB: all-MiniLM-L6-v2, gte-small, bge-small-en/zh-v1.5, multilingual-e5-small, Snowflake Arctic Embed xs/s, jina-embeddings-v2-small-en, mxbai-embed-xsmall-v1), lightweight multilingual (100-400MB: nomic-embed-text-v1.5, bge-m3, EmbeddingGemma-308M, gte-multilingual-base, nomic-v2-moe), and near-production (400MB-1.2GB: mxbai-embed-large-v1, Qwen3-Embedding-0.6B). Key findings: Qwen3-Embedding-0.6B leads Chinese (C-MTEB 71.02) with 32K context and ~400MB Q4 footprint; EmbeddingGemma-308M dominates mobile with <200MB quantized RAM; MRL, long context, and quantization now enable small models to rival much larger ones. Includes a deployment stack overview (int8/int4 quantization, ONNX Runtime, GGUF/llama.cpp, transformers.js v4 with WebGPU) and a scenario-based decision tree for RAG selection.

Key points

This in-depth study (published 2026-07) compares small embedding models for on-device deployment in privacy-first, offline-capable, zero-marginal-cost RAG systems. It evaluates four gating factors: model size on disk, vector dimensions (storage/retrieval cost), context length, and hardware/quantization support.

Benchmark snapshot (MTEB English / C-MTEB Chinese, approximate)

| Model | Params | Dim | Context | Size | MTEB (EN) | C-MTEB | License | |---|---|---|---|---|---|---|---| | all-MiniLM-L6-v2 | 22.7M | 384 | 256 | 46MB | ~56 | ~49 | Apache-2.0 | | gte-small | ~70M | 384 | 512 | 70MB | 61.36 | ~49.5 | Apache-2.0 | | bge-small-en-v1.5 | ~134M | 384 | 512 | 134MB | 62.11 | — | MIT | | bge-small-zh-v1.5 | 118M | 512 | 512 | 139MB | — | 57.82 | MIT | | multilingual-e5-small | ~140M | 384 | 512 | 140MB | 59.93 | 55.38 | MIT | | snowflake-arctic-embed-s | 33M | 384 | 512 | 67MB | retrieval 51.98 | — | Apache-2.0 | | jina-embeddings-v2-small-en | 33M | 512 | 8192 | ~90–130MB | near ada-002 | — | Apache-2.0 | | mxbai-embed-xsmall-v1 | 23M | 384 | 512 | ~90MB | browser-optimized | — | Apache-2.0 | | nomic-embed-text-v1.5 | 137M | 768 (MRL) | 8192 | 274MB | 62.39 | — | Apache-2.0 | | EmbeddingGemma-308M | 308M | 768 (MRL) | 2048 | <200MB RAM (quantized) | 61.15 (multilingual v2) | — | Gemma license | | bge-m3 | 568M | 1024 | 8192 | 1.2GB | ~63 (EN) | 64.8 | MIT | | mxbai-embed-large-v1 | 335M | 1024 | 512 | 670MB | 64.68 | — | Apache-2.0 | | nomic-embed-text-v2-moe | ~305M | 768 | 8192 | 610MB | ~63 | — | Apache-2.0 | | Qwen3-Embedding-0.6B | 0.6B | 1024 (MRL) | 32768 | ~400MB (Q4) | 70.7 (EN v2) | 71.02 | Apache-2.0 |

Tier highlights

Ultra-light (<100MB, browser/Raspberry Pi): all-MiniLM-L6-v2 is the universal default but weak on accuracy and context (256 tokens). gte-small offers +5 MTEB points for 1/3 more size. bge-small-zh-v1.5 is the Chinese pick (C-MTEB 57.82, MIT). multilingual-e5-small covers 100+ languages. Snowflake Arctic Embed xs/s push the size floor. jina-embeddings-v2-small-en uniquely offers 8K context via ALiBi. mxbai-embed-xsmall-v1 is tuned for transformers.js/WebGPU.
Lightweight multilingual (100–400MB): nomic-embed-text-v1.5 (MRL 64–768 dims, 8K context, beats ada-002 at 512 dims with 1/3 the memory) is the default starting point for personal/small-team RAG. bge-m3 outputs dense + sparse (BM25-like) + ColBERT-style multi-vectors in one pass; C-MTEB 64.8. EmbeddingGemma-308M runs in <200MB quantized RAM, <22ms on EdgeTPU, ideal for mobile RAG — but uses the Gemma license, not Apache-2.0. nomic-embed-text-v2-moe (~305M) is a multilingual MoE alternative.
Near-production (400MB–1.2GB): mxbai-embed-large-v1 tops sub-500M English retrieval (64.68) but is capped at 512 tokens. Qwen3-Embedding-0.6B is the standout: 32K context, 100+ languages, instruction-aware, GGUF support; C-MTEB 71.02, NDCG@10 0.782 on Chinese long-document retrieval (vs bge-m3 0.715, nomic-v1.5 0.689) and 0.826 on cross-lingual matching.

Deployment stack

Quantization: int8 dynamic cuts size ~75% with 1–3% accuracy loss and 1.4–2.1× CPU speedup; int4 static halves memory again (needs calibration, suits NPU/DSP). Rule of thumb: 1B params ≈ 1GB int8 / 0.5GB int4.
ONNX Runtime: backends include CPU, CUDA/TensorRT, NNAPI (Android NPU), CoreML (iOS ANE); export via python -m transformers.onnx --model=. --feature=sentence_embeddings onnx/ then quantize_dynamic.
GGUF + llama.cpp: llama-embedding -m model.gguf -p "..." --pooling last runs Qwen3-0.6B easily.
transformers.js v4: WebGPU (~4× faster for BERT-class, C++ rewrite), WASM fallback, fp32/fp16/q8/q4; bge-small hits ~6ms/query in-browser after warm-up. Requires Chrome/Edge 113+, Safari 18+ (WebGPU on iOS 26+).

Decision tree (condensed)

Extremely constrained device (browser cache / RPi / CI): EN → gte-small or all-MiniLM; ZH → bge-small-zh-v1.5; multilingual → multilingual-e5-small
Personal/small-team RAG (laptop CPU): EN → nomic-embed-text-v1.5; ZH/multilingual → Qwen3-Embedding-0.6B (GGUF) or bge-m3
Mobile / offline privacy-first → EmbeddingGemma-308M
English retrieval accuracy on short self-contained docs → mxbai-embed-large-v1
Chinese long-document / cross-lingual / code retrieval → Qwen3-Embedding-0.6B (current best overall pick)

Conclusions

1. Small models are no longer a compromise: Qwen3-0.6B outperforms many larger older models; parameter-count superstition is obsolete. 2. Three technologies define the era: MRL (truncatable dimensions), long context (8K–32K), and quantization (int8/int4). 3. Hybrid dense+sparse retrieval (bge-m3 style) is becoming the RAG norm. 4. Selection rule: fix your scenario first (language, document length, precision, device), then size/dimensions, then benchmark scores — a high score with short context will truncate long documents, and a small model without multilingual training will fail cross-lingual tasks.

> Core insight: if retrieval is poor, even a 70B chat model yields bad answers; if retrieval is good, 7B can win. Invest in "retrieving accurately" before "answering cleverly."

*Scores vary across benchmark versions (v1/v2) and scopes (overall/retrieval/multilingual); compare horizontally with caution. Source compiled from MTEB/MMTEB/C-MTEB leaderboards, HuggingFace model cards, Ollama, Milvus/MorphLLM comparisons, and Google Gemma docs.*