Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search (arXiv 2502.15990)
Overview
This post indexes the arXiv paper Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search (February 2025), filed under the *Verticals* section of a search/recommendation reading list.
| Field | Content | |---|---| | Authors / Affiliations | Jayant Sachdev, Sean D Rosario, Abhijeet Phatak, He Wen, Swati Kirti, Chittaranjan Tripathy | | Published | February 2025 | | Source | https://arxiv.org/abs/2502.15990 | | Type | Academic paper |
Background and Motivation
Large-scale search, recommendation, and personalization systems have long faced challenges in efficiency, scalability, and user intent understanding in information retrieval. Traditional pipeline approaches often treat retrieval, ranking, and generation separately, which struggles to meet LLM-era demands for natural language interaction and multi-hop reasoning. This paper addresses the specific e-commerce search problem of automatically producing query-product relevance labels — historically a labor-intensive manual annotation task — using large language models.
Context from the Reading List
The post situates the work in the intersection of information retrieval and large-scale search/recommendation systems, where a central question is how to redistribute responsibilities among retrieval, ranking, generation, and tool calling in the LLM era.
Typical system decomposition noted in the post:
1. Input and representation: encode queries, documents, and user context as dense or sparse representations, or structured prompts; 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, chained or combined per task; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning, synthetic data bootstrapping; 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Evaluation Considerations
- Datasets: MS MARCO, BEIR, Natural Questions, and domain-specific corpora are common in this space;
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost;
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- An interpretable ensemble of graph and language models for improving search
- Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery (arXiv 2502.10514)
- Behavior Modeling Space Reconstruction for E-Commerce Search (arXiv 2501.18216)
- Behavior-driven query similarity prediction based on pre-trained language models
- Better to Ask in English: Cross-Lingual Evaluation of Large Language Models
- Beyond Relevance: A Demand Balancer Model for Rental Platforms
- Original paper: arXiv:2502.15990
Note: exact experimental numbers should be verified against the original PDF; this post is a metadata-driven digest, not a full translation.
Insights for Search / Rec / Personalization
1. Architecture: cascaded retrieval + rerank + generation remains mainstream, while agentic paradigms make retrieval strategy itself learnable; 2. Data: high-quality instruction data and click/session logs are critical; synthetic data requires protection against leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation; 4. Production: latency, cost, explainability, and safety constraints are hard requirements — academic benchmarks alone are insufficient.