Beyond Benchmarks: Evaluating Embedding Model Similarity for RAG Systems (arXiv 2407.08275)

Forum topic · 小凯 · 2026-07-05

Summary

This arXiv paper (July 2024), authored by Laura Caspari, Kanishka Ghosh Dastidar, Saber Zerhoudi, Jelena Mitrovic, and Michael Granitzer, goes beyond conventional benchmark rankings to study how similarity between embedding models affects Retrieval-Augmented Generation (RAG) systems. Rather than relying solely on aggregate benchmark scores such as nDCG@10 or Recall@k on datasets like BEIR and MS MARCO, the work examines whether two embedding models behave similarly, and what that implies for transferring retrieval results, model selection, and evaluation trustworthiness in RAG pipelines. The post situates the paper within the broader evolution from BM25 and dense bi-encoders to cross-encoders, late interaction, generative retrieval, and agentic search, and discusses engineering considerations: latency and cost budgets, embedding version management, index refresh strategies, hybrid sparse-dense retrieval, hallucination and security risks, and the gap between offline metrics and online user satisfaction. It also cross-references related embedding work including BGE M3, Arctic-Embed 2.0, and BGE-en-ICL. Readers evaluating or selecting embedding models for production RAG should consult the original PDF for exact experimental tables and quantitative conclusions.

Beyond Benchmarks: Evaluating Embedding Model Similarity for RAG Systems (arXiv, July 2024)

Authors: Laura Caspari, Kanishka Ghosh Dastidar, Saber Zerhoudi, Jelena Mitrovic, Michael Granitzer
Source: https://arxiv.org/abs/2407.08275
Type: Academic paper — Embedding models

Overview

The paper argues that standard benchmark leaderboards are insufficient for choosing embedding models for Retrieval-Augmented Generation (RAG). Instead of only comparing models by aggregate retrieval metrics, it analyzes the similarity between embedding models themselves — i.e., whether different models retrieve the same or different documents — and what this means for model selection, result transferability, and the reliability of benchmark-based conclusions in RAG pipelines.

Key Points

Benchmark scores alone may hide substantial behavioral differences between embedding models; two models with close nDCG@10 can retrieve noticeably different document sets.
Embedding-model similarity analysis can inform whether retrieval results from one model can substitute or complement another in a RAG stack.
The work situates itself in the modern retrieval stack: dense bi-encoders, cross-encoder rerankers, late-interaction models, and LLM-centric agentic retrieval.

Positioning in the Field

Neural IR has evolved from BM25 to dense retrieval, cross-encoder reranking, and now generative/agentic search where the LLM decides *whether and how often* to retrieve. In this landscape, RAG evaluation is shifting from static nDCG toward task success, citation accuracy, and multi-hop reasoning quality — making embedding model behavior, not just its leaderboard score, a critical variable.

Practical Takeaways for Engineers

| Concern | Recommendation | |---------|----------------| | Latency | Cascaded retrieval + early stopping, cache hot queries | | Quality | Verify offline gains transfer to online CTR/satisfaction via interleaving tests | | Data safety | PII partitioning, versioned embeddings, rollback capability | | Security | Source whitelisting, poisoning detection, output filtering | | Cost | Hybrid sparse+dense retrieval, model routing/distillation |

Common Evaluation Setups Referenced in This Space

Datasets: BEIR, MS MARCO, Natural Questions, domain-specific corpora
Metrics: nDCG@10, MRR, Recall@k, Hit@k, task success rate
Baselines: BM25, dense retrieval, cross-encoder rerankers, retrieval-free LLMs

Related Entries

BGE M3-Embedding (arXiv:2402.03216)
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
BGE-en-ICL / BGE-ICL few-shot embedders (arXiv:2409.15700)

> Note: Exact experimental tables and quantitative results should be verified against the original PDF at arXiv:2407.08275 before citing numbers.