LLM-Driven Usefulness Judgment for Web Search Evaluation
Paper: https://arxiv.org/abs/2504.14401 (arXiv, April 2025)
Authors: Mouly Dewan, Jiqun Liu, Aditya Gautam, Chirag Shah
Overview
This paper examines whether large language models (LLMs) can be used to generate usefulness judgments for web search evaluation. Classical evaluation of search systems depends on human relevance judgments, which are expensive to collect and measure *relevance* rather than *usefulness*—that is, whether a document genuinely helps a user accomplish their underlying task.
Key Points
- Motivation: Relevance judgments (e.g., graded relevance feeding nDCG-style metrics) do not fully capture whether a result actually serves a user's search task. Usefulness judgments are more user-centric but costly to gather from human assessors.
- Approach: The authors leverage LLMs as automated judges to assess the usefulness of web documents in the context of a user's information need, following the broader "LLM-as-a-judge" paradigm.
- Evaluation concern: The work studies the alignment between LLM-generated usefulness judgments and human judgments, and how LLM-based usefulness evaluation compares with traditional relevance-based evaluation.
- Context: The paper contributes to ongoing efforts to reduce the cost and increase the scalability of search evaluation while shifting metrics closer to actual user benefit.
- Search evaluation has long been criticized for the gap between offline relevance metrics and real user satisfaction. Automating usefulness judgments with LLMs could make user-centric evaluation practical at scale.
- It connects to larger trends in IR: generative retrieval, RAG, and agentic search all require evaluation methods that go beyond static relevance labels toward task success and utility.
- The approach raises open questions typical of LLM-judge pipelines: reliability, bias, cost/latency, and the need for cross-validation against human assessment.
- Original paper: LLM-Driven Usefulness Judgment for Web Search Evaluation
Why It Matters
Limitations
As with most LLM-as-judge work, potential concerns include consistency across runs, sensitivity to prompt design, domain and language coverage, and disagreement between LLM and human notions of usefulness. Quantitative details, datasets, and agreement statistics are available in the full paper PDF; readers should consult the original before citing specific numbers.