SIGIR 2024: First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval

Forum topic · 小凯 · 2026-07-05

Summary

This post introduces the First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval (LLM4Eval), held at SIGIR 2024. The workshop examines how LLMs can transform IR evaluation, including LLM-as-a-judge relevance assessment, synthetic query and judgment generation, and evaluation of retrieval-augmented generation (RAG) and agentic search systems. The post situates the workshop within the broader evolution of neural IR—from BM25 and dense retrieval to generative retrieval—arguing that evaluation must shift from static metrics like nDCG toward process-oriented measures such as task success rate and citation accuracy. It outlines a typical four-stage methodology (input representation, core modules, learning strategy, inference pipeline), lists open problems such as evaluation trustworthiness, latency, cost, hallucination, and cross-lingual generalization, and provides an engineering checklist covering data privacy, latency budgets, quality auditing, safety filtering, and cost control. The entry links to the official workshop papers page and cross-references related events including SIGIR 2025 eCommerce, CIKM 2024 Multimodal Search, and EACL 2024 Generative AI personalization workshops.

SIGIR 2024: The First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval

Overview

This entry describes the First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval (LLM4Eval), co-located with SIGIR 2024. The workshop focuses on how LLMs are reshaping evaluation methodology in information retrieval: using LLMs as relevance judges, generating synthetic queries and judgments, and benchmarking LLM-centric retrieval and RAG pipelines.

Official papers page: https://llm4eval.github.io/SIGIR2024/papers/
Resource type: Conference / Workshop
Section: Conferences, Workshops

Background and Scope

Large-scale search, recommendation, and personalization systems have long faced challenges in efficiency, scalability, and user-intent understanding. Traditional pipelines separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. This workshop addresses the intersection of LLMs and IR evaluation, covering open-domain search, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation, and end-to-end architectures combining external knowledge with generative models.

Key Themes and Contributions

A unified perspective that brings scattered evaluation work into a comparable framework.
Clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
Reproducible benchmarks, datasets, and taxonomies that lower the entry cost for follow-up research.
Interfaces to emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, with migration paths from research prototypes to industrial systems.
Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.

Methodological Patterns

Work in this area typically follows a four-step pattern:

1. Input and representation: encode queries, documents, and user context into dense/sparse representations or structured prompts. 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, composed serially or in parallel. 3. Learning strategy: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrapped data synthesis. 4. Inference strategy: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.

Implications for Search and Recommendation

1. Architecture: cascaded retrieval + rerank + generation remains mainstream, but agentic paradigms make "when and how often to retrieve" a learnable decision. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated against human assessment. 4. Product: latency, cost, interpretability, and safety are hard constraints for industrial deployment—not just academic benchmark scores.

Engineering Checklist

| Check | Question | Recommendation | |-------|----------|----------------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-safe embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stopping, query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Tokens and GPU usage per query? | Small-model routing, distillation, hybrid sparse+dense |

Limitations

Possible limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric data leaving cross-lingual generalization unknown, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.

Related Entries

2025 SIGIR Workshop on eCommerce
CIKM 2024 1st Workshop on Multimodal Search and Recommendations
EACL 2024 Workshop on Personalization of Generative AI Systems
ICDM MMSR 2025

References

Original source: SIGIR 2024 The First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval. See the official workshop page for publication details.