SIGIR 2024 First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval
Overview
This entry catalogs the First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval, held at SIGIR 2024. The official program and accepted papers are listed at: https://llm4eval.github.io/SIGIR2024/papers/
| Field | Content | |-------|---------| | Type | Conference workshop | | Venue | SIGIR 2024 | | Link | https://llm4eval.github.io/SIGIR2024/papers/ | Category | Conferences, Workshops |
Background and Scope
Information retrieval systems at scale have long faced challenges in efficiency, scalability, and user intent understanding. Traditional pipeline approaches separate retrieval, ranking, and generation, which struggles to meet modern demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. This workshop was organized to systematically advance the theory and practice of using LLMs for evaluation in this intersection area.
Core scenarios addressed by the workshop papers include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation, and end-to-end architectures combining external knowledge sources with generative models.
Key Themes
- A unified perspective on LLM-based evaluation, bringing scattered work into a comparable framework.
- Clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
- Reproducible benchmarks, datasets, and taxonomies that lower the entry barrier for subsequent researchers.
- Interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, with migration paths from research prototypes to industrial systems.
- Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal extension.
- 2025 SIGIR Workshop on eCommerce
- CIKM 2024 1st Workshop on Multimodal Search and Recommendations
- EACL 2024 Workshop on Personalization of Generative AI Systems
- ICDM MMSR 2025
- Original: SIGIR 2024 First Workshop on LLMs for Evaluation in Information Retrieval. See workshop paper listing.
Typical Method Pipeline
1. Input and representation: encode queries, documents, and user context into dense/sparse representations or structured prompts. 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, chained or run in parallel per task. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrap data synthesis. 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Insights for Search, Recommendation, and Personalization
1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm is making "retrieval count and policy" itself a learnable object. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human assessment. 4. Product: latency, cost, interpretability, and safety policies are hard constraints for industrial deployment — academic benchmarks alone are not enough.
Engineering Checklist
| Item | Question | Suggestion | |------|----------|------------| | Data | Does training/index data contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stop, cache popular queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Small-model routing, distillation, hybrid sparse+dense retrieval |
Limitations
Possible limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, unknown cross-lingual generalization from English-centric data, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |