Report on LLM4Eval 2024: The 1st Workshop on LLMs for Evaluation in Information Retrieval at SIGIR 2024

Forum topic · 小凯 · 2026-07-05

Summary

This post indexes the official report on LLM4Eval 2024, the first Workshop on Large Language Models for Evaluation in Information Retrieval, held at SIGIR 2024. The workshop examines how large language models can be used for evaluation tasks in information retrieval, including LLM-as-judge relevance assessment, test collection construction, and evaluation methodologies for LLM-era search systems. The source is published in the ACM Digital Library under the SIGIR proceedings (DOI: 10.1145/3722449.3722461) and is categorized under search engine evaluation. The surrounding analysis places the work in the context of modern retrieval-augmented generation (RAG), agentic search, and learning-to-rank pipelines, discussing how evaluation is shifting from static metrics such as nDCG toward process-oriented measures like task success rate and citation accuracy. Open challenges include evaluation trustworthiness, latency and cost constraints, hallucination and safety risks, and cross-lingual and multimodal extension. Note that only minimal metadata and abstract text are available for the original report, so quantitative findings should be verified against the full PDF.

Report on LLM4Eval 2024: The 1st Workshop on Large Language Models for Evaluation in Information Retrieval at SIGIR 2024

Overview

This post indexes the workshop report "Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024)", published in the ACM Digital Library as part of the SIGIR 2024 proceedings.

Source: https://dl.acm.org/doi/abs/10.1145/3722449.3722461
Venue: SIGIR 2024
Topic area: Evaluation of Search Engines
Type: Academic workshop report

Context: LLMs and IR Evaluation

The LLM4Eval workshop addresses how large language models (LLMs) are reshaping evaluation methodology in information retrieval (IR). Traditional IR evaluation relies on human-labeled relevance judgments and static test collections built around metrics such as nDCG@10 and MRR. LLMs introduce new possibilities and new problems:

LLM-as-judge: using LLMs as automatic relevance assessors, which requires calibration against human judgments to ensure trustworthiness.
Evaluating LLM-based systems: search and recommendation pipelines increasingly include retrieval-augmented generation (RAG) and agentic behaviors, demanding process-level metrics (task success rate, citation accuracy, multi-hop reasoning completeness) rather than purely ranking-based scores.
Cost, latency, and reproducibility: practical evaluation protocols must account for inference budgets and model version drift.

Key Takeaways for the Community

1. Architecture: cascaded retrieve-then-rerank-then-generate remains mainstream, but agentic paradigms make retrieval strategy itself a learnable decision. 2. Data: high-quality instruction data and click/session logs matter; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online user satisfaction is widening; LLM-as-judge should be cross-validated with human assessment. 4. Deployment: latency, cost, interpretability, and safety are hard constraints for production systems, not just benchmark optimization targets.

Related Entries

Evaluation of Retrieval-Augmented Generation: A Survey
ARES: An Automated Evaluation Framework for RAG
AI Search Has A Citation Problem (CJR, Mar 2025)
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation |

Notes

Only minimal metadata and abstract text are available for the original workshop report in this entry. Readers citing quantitative conclusions should consult the full PDF at the ACM Digital Library link above.

> Original abstract: "Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024."

References

Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024, SIGIR. https://dl.acm.org/doi/abs/10.1145/3722449.3722461