Automated Query-Product Relevance Labeling Using Large Language Models for E-commerce Search
Overview
- Paper: Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search
- Authors: Jayant Sachdev, Sean D Rosario, Abhijeet Phatak, He Wen, Swati Kirti, Chittaranjan Tripathy
- Published: February 2025, arXiv
- Link: https://arxiv.org/abs/2502.15990
- Train and fine-tune retrieval and ranking models
- Augment or replace scarce human-annotated data
- Enable rapid iteration when catalog content or query distributions change
- LLM-as-annotator pipelines are becoming a standard component of industrial search data infrastructure.
- Agreement with human labels must be validated before replacing manual annotation wholesale; hybrid human-LLM labeling is a common compromise.
- Offline label quality should still be cross-checked against online metrics (CTR, satisfaction), as label-level agreement does not guarantee end-to-end gains.
- Behavior Modeling Space Reconstruction for E-Commerce Search (arXiv:2501.18216)
- Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery (arXiv:2502.10514)
- Behavior-driven query similarity prediction based on pre-trained language models
Background and Motivation
E-commerce search systems depend on large volumes of query-product relevance labels for training retrieval and ranking models. Traditionally, these labels are produced by human judges — a process that is slow, expensive, hard to scale, and prone to inconsistency across annotators. As product catalogs and query volumes grow, manual labeling becomes a serious bottleneck.
This paper investigates whether large language models (LLMs) can automate this labeling process, generating graded relevance judgments for query-product pairs at scale while maintaining alignment with human judgment.
Core Idea
The authors propose using LLMs to assess the relevance of a product to a given shopping query, producing graded relevance labels that can be used to:
Key considerations include prompt design for e-commerce relevance (understanding user intent, product attributes, and match quality) and calibrating LLM outputs against human-labeled benchmarks.
Why It Matters
1. Cost and scale: LLM labeling can reduce the cost of relevance annotation by orders of magnitude compared to human judges. 2. Consistency: Models apply criteria uniformly, avoiding drift between annotators. 3. Data flywheel: Better labels improve retrieval/ranking models, which improve search quality and downstream conversion.
Takeaways for Search/Rec Practitioners
> Note: This post is based on the paper's abstract and public metadata. For exact experimental results, datasets, and methodology details, refer to the original paper on arXiv.