BGE-en-ICL: Making Text Embedders Few-Shot Learners (arXiv 2409.15700, Sep 2024)
This forum entry documents the paper "Making Text Embedders Few-Shot Learners" by Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, et al. (8 authors), available at https://arxiv.org/abs/2409.15700.
Key idea
Most embedding models are frozen at deployment and cannot benefit from a handful of labeled examples. BGE-en-ICL makes embedding models few-shot learners through in-context learning (ICL): instead of updating model weights, the query is augmented with a small number of input–output demonstration pairs formatted as structured context. The model then produces task-aware embeddings, improving retrieval, classification, clustering, and semantic textual similarity (STS) performance.
Contributions
- Introduces BGE-en-ICL, an English embedding model that supports both standard (zero-shot) encoding and query-side in-context learning.
- Proposes a decoupled contrastive learning strategy: training separates the representations of the query itself and the augmented ICL context, so few-shot capability does not sacrifice zero-shot performance.
- Demonstrates gains on MTEB-style evaluation benchmarks when few-shot examples are provided, with the same model remaining competitive when examples are omitted.
- Shows that ICL lets one embedding model adapt to diverse downstream tasks without task-specific fine-tuning, similar in spirit to how LLMs use prompts.
- Evaluation across embedding benchmark suites covering retrieval, STS, classification, and clustering tasks.
- Baselines include the standard BGE model and other strong dense embedders; ICL augmentation brings measurable improvements across task types.
- The post notes that exact figures should be verified against the original PDF tables.
- BGE M3-Embedding (arXiv:2402.03216) — multi-lingual, multi-functionality, multi-granularity embedding.
- Arctic-Embed 2.0 (arXiv:2412.04506) — multilingual retrieval.
- Scandinavian Embedding Benchmarks — multilingual evaluation.
- Beyond Benchmarks (arXiv:2407.08275) — evaluating embedding model similarity.
Experimental setup (per the paper)
Context within the embedding ecosystem
The post cross-references related entries:
Engineering takeaways
1. Architecture: query-side ICL is a low-cost adaptation path compared with fine-tuning or LoRA; no index rebuild is needed since only queries carry context. 2. Data: few-shot examples must be representative and free of leakage; quality of demonstrations directly drives gains. 3. Evaluation: report both zero-shot and few-shot modes; consistent degradation checks are needed to confirm decoupling works. 4. Deployment: added prompt tokens increase query encoding latency slightly; caching and prompt-length budgeting are recommended.
Limitations and future work
The approach is currently demonstrated for English; extending ICL-style adaptation to multilingual and instruction-following embedders, and studying how many demonstrations are optimal per task, remain open directions.
> Original abstract (as listed): BGE-en-ICL, BGE-ICL embedding model, Making Text Embedders Few-Shot Learners, Sep 2024, arxiv — https://arxiv.org/abs/2409.15700