Large Language Models for Relevance Judgment in Product Search (arXiv 2406.00247)

Forum topic · 小凯 · 2026-07-05

Summary

This forum post indexes the arXiv paper "Large Language Models for Relevance Judgment in Product Search" (arXiv:2406.00247), authored by Navid Mehrdad, Hrushikesh Mohapatra, Mossaab Bagdouri, Prijith Chandran, Alessandro Magnani, Xunfan Cai, and colleagues (11 authors in total). The paper explores using large language models (LLMs) to perform or assist relevance judgment in e-commerce product search, a task that traditionally requires expensive and slow human annotation of query-product pairs. The surrounding post frames the work within the evaluation-of-search-engines section of an IR/LLM reading list, and provides general context on how LLMs are reshaping retrieval, ranking, and evaluation pipelines: cascade architectures, LLM-as-judge evaluation, cost/latency constraints, and open problems such as evaluation trustworthiness and hallucination. Note that the post itself is largely a template with a placeholder abstract, so quantitative results should be verified against the original PDF.

Large Language Models for Relevance Judgment in Product Search

Source: arXiv:2406.00247 — indexed under "Evaluation of Search engines"

Authors: Navid Mehrdad, Hrushikesh Mohapatra, Mossaab Bagdouri, Prijith Chandran, Alessandro Magnani, Xunfan Cai, et al. (11 authors in total)

One-line summary

The paper investigates how large language models can be used for relevance judgment in product search, where traditional human annotation of query-product relevance is costly and slow.

Context

This post appears in an IR/LLM reading list. Relevance judgment is a core bottleneck in search evaluation: collecting labeled query-document (here, query-product) pairs at scale requires extensive human effort. LLMs have recently been proposed as scalable judges ("LLM-as-judge"), raising questions about agreement with human assessors, cost, and bias.

Key points from the surrounding discussion

The work sits at the intersection of information retrieval and large-scale search/recommendation systems, addressing how retrieval, ranking, generation, and evaluation responsibilities shift in the LLM era.
Typical methodological pipeline for such systems: input/representation of queries and products, core modules (retriever, reranker, judge), learning strategies (SFT, distillation, RLHF-style signals), and inference strategies (single-pass or iterative retrieval with budget control).
Common evaluation protocols in this space use datasets and metrics such as nDCG@10, MRR, Recall@k, compared against baselines like BM25, dense retrieval, and cross-encoder rerankers.
Open problems highlighted: trustworthiness of LLM judges, latency and token cost, hallucination and safety, cross-lingual and multimodal generalization.
For practitioners: check index refresh cadence, embedding version compatibility, online A/B sensitivity, and failure modes (empty retrieval, wrong judgments) before deploying LLM-based relevance judgments.

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | LLM-as-judge | Using an LLM to evaluate relevance or quality in place of human assessors | | Agentic Search | Modeling search as sequential decision-making and tool use |

Note on fidelity

The source forum post is mostly a template with a placeholder abstract (the abstract field repeats the title). Quantitative findings and specific experimental results should be verified against the original paper PDF at https://arxiv.org/abs/2406.00247.

Related entries

Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
A Dataset of Information-Seeking Questions and Answers Anchored in Research Articles (arXiv:2105.03011)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)