NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (NVIDIA, May 2024)

NV-Embed is an embedding model from NVIDIA, based on Mistral-7B, that introduces architectural and training techniques for turning large language models into strong generalist embedding models. The paper is available at https://arxiv.org/abs/2405.17428.

Key contributions

• Latent attention architecture: NV-Embed proposes a novel latent attention layer that produces pooled sentence-level embeddings, replacing the conventional mean pooling or last-token pooling used by previous LLM-based embedding models.
• Contrastive learning with large batches: The model is trained with contrastive learning using a large batch size, with harder hard negatives and a two-stage instruction-tuning contrastive training method that improves accuracy on non-retrieval tasks without degrading retrieval performance.
• State-of-the-art results: At the time of publication, NV-Embed achieved a new state of the art on the MTEB benchmark with an average score of 56.32 across 56 tasks, and ranked first on the MIRACL retrieval benchmark with a Recall@100 of 5.59.

Authors

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and colleagues (7 authors total) from NVIDIA.

Context in the embedding literature

The post places NV-Embed alongside related embedding research indexed in the same collection:

• The Scandinavian Embedding Benchmarks — multilingual embedding evaluation
• A Universal Framework for Compressing Embeddings in CTR Prediction
• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity
• BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval

Takeaways

• Decoder-only LLMs can serve as highly effective generalist embedders when paired with appropriate pooling (latent attention) and training (instruction-tuned contrastive learning).
• The two-stage training recipe shows that non-retrieval accuracy and retrieval accuracy do not have to trade off against each other.
• Readers should consult the paper PDF for exact experimental tables and quantitative details, as this post summarizes the abstract-level findings.