Enhancing Relevance of Embedding-based Retrieval at Walmart (CIKM 2024)

This post catalogs an applied research paper presented at CIKM 2024 (October 2024) from Walmart, listed in the ACM Digital Library:

• Source: https://dl.acm.org/doi/abs/10.1145/3627673.3680047
• Type: Academic paper (industry track)
• Topic area: Embedding-based retrieval for large-scale e-commerce search

> Note: The original forum post is a metadata/catalog entry in an awesome-list style collection. Its abstract and results sections are placeholders — the substantive content below summarizes the post's reading framework and context rather than verified paper results. Consult the ACM page for the authoritative abstract and experiments.

Key points

• Problem domain: Embedding-based retrieval at industrial scale, where relevance, efficiency, and scalability must be balanced in production search systems.
• Paper link: https://dl.acm.org/doi/abs/10.1145/3627673.3680047 — the post does not reproduce quantitative results; verify all numbers against the original PDF.
• Typical pipeline framing offered by the post for such systems:
1. Encode queries, documents, and user context into dense or sparse representations. 2. Core modules: retriever, reranker, planner, memory, tool interfaces. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning, synthetic data. 4. Inference strategies: single-pass vs. iterative retrieval, parallel sub-queries, early stopping and budget control.
• Evaluation practice the post recommends: report nDCG@10, MRR, Recall@k alongside latency and token cost; cross-validate LLM-as-judge with human evaluation; check whether offline gains translate into online CTR/satisfaction.

Engineering checklist from the post

| Check | Question | Suggestion | |-------|----------|------------| | Data | PII in training/index? Version control? | Partitioned index, anonymization, rollback-able embedding versions | | Latency | p99 budget? Retrieval steps? | Cascaded retrieval + early stopping, query caching, async reranking | | Quality | Does offline gain convert to online CTR? | Interleaving experiments, human audits, citation checks | | Safety | Retrieval poisoning / bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Route to small models, distillation, hybrid sparse+dense |

Context and related entries

The post situates embedding-based retrieval within the evolution from BM25 → dense dual-tower retrieval → cross-encoder reranking → generative retrieval → RAG and agentic search. It cross-references sibling entries including:

• Automated Query-Product Relevance Labeling using LLMs
• Behavior-driven query similarity prediction based on pre-trained language models
• Behavior Modeling Space Reconstruction for E-Commerce Search
• An interpretable ensemble of graph and language models

Takeaways for practitioners

1. Researchers: replicate core baselines; check for statistical significance and compute-cost reporting. 2. Engineers: extract pluggable modules (encoder, reranker, planner) and assess integration cost with existing stacks. 3. Product managers: focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency), not just offline nDCG.

Glossary

• IR — Information Retrieval
• RAG — Retrieval-Augmented Generation
• LTR — Learning to Rank
• nDCG — Normalized Discounted Cumulative Gain
• Agentic Search — modeling search as sequential decision-making and tool invocation
• Gen-IR — Generative Information Retrieval