Large Language Models for Relevance Judgment in Product Search (arXiv 2406.00247)
Note: This forum post is primarily an index/catalog entry. Much of its body consists of generic commentary templates rather than a detailed reproduction of the paper, so some sections below are summarized cautiously. Quantitative findings should be verified against the original PDF.
Metadata
| Field | Value | |-------|-------| | Title | Large Language Models for Relevance Judgment in Product Search | | Authors | Navid Mehrdad, Hrushikesh Mohapatra, Mossaab Bagdouri, Prijith Chandran, Alessandro Magnani, Xunfan Cai, et al. (11 authors) | | Published | July 2024 (arXiv listing, June 2024 ID) | | Source | https://arxiv.org/abs/2406.00247 | | Type | Academic paper | | Category | Evaluation of Search Engines |
What the Paper Is About
The paper studies the use of large language models (LLMs) as relevance judges in product search evaluation. Relevance judgment — deciding whether a retrieved product actually matches a user's query intent — has traditionally relied on costly, slow human annotation. LLM-based judgment (LLM-as-judge) promises scalable, reproducible alternatives, but raises questions about agreement with human assessors, calibration, and domain fit in e-commerce settings.
Context from the Post
The surrounding commentary situates the work within LLM-era information retrieval:
- Architecture trends: cascaded retrieve → rerank → generate remains mainstream; agentic paradigms make retrieval strategies themselves learnable.
- Evaluation practices: standard datasets (MS MARCO, BEIR, Natural Questions) and metrics (nDCG@10, MRR, Recall@k), with a noted gap between offline metrics and online user satisfaction — LLM-as-judge should be cross-validated with human assessment.
- Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal extension.
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles (arXiv:2105.03011)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
- AI Search Has A Citation Problem (CJR, March 2025)
Engineering Checklist (from the post)
| Concern | Recommendation | |---------|----------------| | Data | PII partitioning, sanitization, versioned embeddings | | Latency | Cascade with early stopping, query caching, async reranking | | Quality | Interleaving experiments, human audit samples, citation checks | | Safety | Source allowlisting, adversarial detection, output filtering | | Cost | Small-model routing, distillation, hybrid sparse+dense retrieval |
Caveats
The post's own one-line summary and abstract-translation slots are left as template placeholders, and its experimental sections are generic. Specific numbers from the paper (agreement rates with human judgments, model comparisons, dataset details) are not reproduced in the post and must be read from the original arXiv page.