Report on the 1st Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
This is an English-language presentation of a Chinese forum post covering the workshop report for LLM4Eval 2024, the 1st Workshop on Large Language Models for Evaluation in Information Retrieval, held at SIGIR 2024.
Overview
- Title: Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
- Venue: SIGIR 2024
- Source: https://dl.acm.org/doi/abs/10.1145/3722449.3722461
- Topic area: Evaluation of search engines
- Context: Traditional IR pipelines separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural language interaction, multi-hop reasoning, and real-time knowledge. LLM4Eval addresses evaluation methodology for this new landscape.
- Methodological framing: Modern systems follow a "problem formulation → model/system design → training/construction → inference pipeline" pattern, with components including retrievers, rerankers, planners, memory modules, and tool interfaces, and learning strategies spanning supervised fine-tuning, contrastive learning, distillation, and reinforcement learning.
- Common evaluation protocols in this space include:
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain
- Agentic Search: Modeling search as sequential decision-making and tool use
- Gen-IR: Generative Information Retrieval
- Evaluation of Retrieval-Augmented Generation: A Survey (arXiv:2405.07437)
- ARES: An Automated Evaluation Framework for RAG (arXiv:2311.09476)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (arXiv:2401.13178)
- AI Search Has A Citation Problem (CJR, March 2025)
The workshop focuses on the intersection of large language models and information retrieval evaluation, covering how LLMs can serve as judges, annotators, and evaluation infrastructure for search and recommendation systems.
Key points
Insights for search, recommendation, and personalization
1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms are making retrieval count and strategy learnable objects. 2. Data: High-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human evaluation. 4. Productization: Latency, cost, explainability, and safety are hard constraints for industrial deployment—academic benchmarks alone are insufficient.
Engineering checklist (from the source post)
| Item | Question | Recommendation | |---|---|---| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? How many retrieval steps? | Cascades + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, manual audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense |