LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? (arXiv, Jan 2025)

Metadata

| Field | Value | |------|------| | Title | LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? | | Authors | Rikiya Takehi, Ellen M. Voorhees, Tetsuya Sakai, Ian Soboroff | | Published | January 2025 (arXiv) | | Link | https://arxiv.org/abs/2411.06877 | | Type | Academic paper | | Topic area | Evaluation of search engines / information retrieval |

Overview

Relevance assessments are the backbone of information retrieval (IR) evaluation, but having human assessors judge query-document pairs is expensive and slow at scale. This paper examines the use of large language models (LLMs) to assist or substitute for human relevance judgments, and analyzes when it is appropriate to ask LLMs for help — i.e., under which conditions LLM-based assessments can be trusted to support evaluation outcomes.

Context

The paper sits at the intersection of IR evaluation and the LLM era. Key pressures motivating this work include:

• Traditional test-collection methodology (human assessors pooling documents and judging relevance) does not scale to modern corpus sizes and rapid iteration cycles.
• LLM-as-judge approaches promise cheap, fast labeling, but raise questions about reliability, bias, and agreement with human judgments.
• Evaluation credibility matters downstream: ranked leaderboards, RAG pipelines, and retrieval benchmarks all depend on the quality of relevance labels.

Key points

• The work systematically studies LLM-assisted relevance assessment rather than adopting LLM judgments wholesale.
• It treats the LLM as an assistant whose outputs must be selectively trusted, analyzing when LLM labels are dependable enough to influence evaluation conclusions.
• The problem is framed around the recurring trade-offs in modern IR evaluation: label quality vs. cost, human vs. model judgment, and the risk that flawed assessments propagate into misleading system rankings.
• Related open problems flagged in this space include evaluation trustworthiness, hallucination and bias in automated judgments, and the need to cross-validate LLM-as-judge outputs against human assessments.

Engineering takeaways

For teams using LLM judges in production evaluation pipelines:

1. Cross-validate LLM judgments with sampled human assessments before trusting aggregate metrics. 2. Budget-aware design: route easy judgments to smaller/cheaper models and reserve human effort for ambiguous cases. 3. Audit failure modes: empty retrievals, systematic bias toward authoritative-sounding text, and position or verbosity biases in pairwise judging. 4. Report uncertainty: treat LLM-derived relevance labels as probabilistic signals, not ground truth.

> Note: This page is a metadata and contextual summary. Verify quantitative results against the original PDF at https://arxiv.org/abs/2411.06877.

Related entries

• Evaluation of Retrieval-Augmented Generation: A Survey
• ARES: An Automated Evaluation Framework for RAG
• AI Search Has A Citation Problem (CJR, Mar 2025)

Glossary

| Term | Meaning | |------|------| | IR | Information Retrieval | | Relevance assessment | Human or model judgment of how well a document answers a query | | LLM-as-judge | Using LLMs to automatically evaluate outputs or labels | | Test collection | Shared set of topics, documents, and relevance judgments used for IR evaluation |