Marqo Ecommerce Embedding Benchmarks: Text-to-Image and Category-to-Image Tasks on Hugging Face
Overview
Marqo maintains a public benchmark space on Hugging Face dedicated to evaluating embedding models for eCommerce applications:
- Resource: Marqo/Ecommerce-Embedding-Benchmarks
- Task families: text-to-image retrieval and category-to-image retrieval in product search contexts
- Domain: eCommerce / product-centric search, rather than general web text
- A leaderboard comparing embedding models on eCommerce retrieval tasks
- Text-to-image evaluation: how well query text retrieves relevant product images
- Category-to-image evaluation: how well category labels align with product images
- A reproducible setup for practitioners selecting models for product search, recommendation, and visual discovery pipelines
- General multilingual and retrieval benchmarks (e.g., BGE M3-Embedding, Arctic-Embed 2.0)
- Embedding similarity analyses that look beyond raw benchmark scores
- Embedding compression work for large-scale retrieval systems
- Leaderboard contents and rankings change over time; consult the Hugging Face space for current results.
- Specific metric values and dataset details should be verified against the original resource rather than secondary summaries.
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual Embeddings
- A Universal Framework for Compressing Embeddings in CTR Prediction
- Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
- BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity
- BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners
- Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval
Why an eCommerce-specific benchmark?
General embedding benchmarks (e.g., MTEB-style leaderboards) are dominated by web text, Wikipedia, and generic image datasets. Product search differs in several ways:
1. Query style: user queries are short, attribute-heavy phrases (brand, color, size, use case) rather than natural-language sentences. 2. Multimodal matching: the target of a search is often a product image, requiring cross-modal alignment between text and product photography. 3. Category structure: mapping category labels to product images tests whether models capture taxonomy-level semantics, not just surface similarity.
Models that perform well on generic benchmarks may underperform on these tasks, which motivates a domain-specific evaluation suite.
What the benchmark provides
Positioning among related work
This benchmark complements broader embedding evaluation efforts, including:
For teams building search over product catalogs, a domain-specific leaderboard reduces the risk of picking a model that generalizes well on paper but fails on real product queries.