LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? (arXiv, Jan 2025)

Overview

This paper, available at https://arxiv.org/abs/2411.06877, is authored by Rikiya Takehi, Ellen M. Voorhees, Tetsuya Sakai, and Ian Soboroff. It falls under the topic of Evaluation of Search Engines in information retrieval (IR).

Research Background

Relevance assessments — human judgments of whether a document is relevant to a query — are the cornerstone of search engine evaluation and reusable test collections. However, manual assessment is expensive, slow, and does not scale to the volume of modern retrieval systems. With the rise of large language models (LLMs), using LLMs as automatic relevance judges has become an active research direction, raising the question the paper's title asks: when should we actually ask LLMs for help with relevance assessment?

Key Points

• The paper examines the use of LLMs to assist in the relevance assessment process for IR evaluation, rather than replacing human assessors wholesale.
• It addresses the reliability of LLM-generated judgments compared to human judgments, a prerequisite for trustworthy evaluation results.
• It contributes to the broader "LLM-as-judge" discussion, including concerns about evaluation credibility, bias, and the need for cross-validation between automatic and human assessment.

Context for Practitioners

The forum discussion around this paper situates it within LLM-era search evaluation:

• Architecture: Cascaded retrieve-rank-generate pipelines remain mainstream, while agentic approaches make retrieval strategy itself a learned decision.
• Evaluation: The gap between offline metrics (e.g., nDCG, MRR, Recall@k) and online user satisfaction is widening; LLM judges need to be checked against human evaluation.
• Engineering constraints: Latency, cost, explainability, and safety remain hard requirements for production systems and cannot be traded away for benchmark gains.

Limitations

Common limitations in this research area include experiments bounded by compute budgets, benchmarks that may not match real user distributions, and English-centric data that leaves cross-lingual generalization open. Readers should consult the PDF for exact experimental numbers before citing quantitative conclusions.

Related Work

• Evaluation of Retrieval-Augmented Generation: A Survey (arXiv 2405.07437)
• ARES: An Automated Evaluation Framework for RAG (arXiv 2311.09476)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Abstracts (arXiv 2105.03011)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | LLM-as-judge | Using an LLM in place of human evaluators for assessment tasks | | Reusable test collections | Shared query-document-judgment resources enabling comparable IR experiments |