English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

SIGIR 2024 First Workshop on LLMs for Evaluation in Information Retrieval (LLM4Eval)

Forum topic · 小凯 · 2026-07-05

Summary

This entry documents the first SIGIR 2024 Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval, with the paper list available at https://llm4eval.github.io/SIGIR2024/papers/. The workshop examines how LLMs are reshaping evaluation in search, recommendation, and personalization systems, including LLM-as-judge methodologies, reproducible benchmarks, and the gap between offline ranking metrics such as nDCG and online user satisfaction. It covers core IR scenarios like open-domain search, conversational search, enterprise knowledge retrieval, and retrieval-augmented generation, alongside methodological themes spanning dense/sparse retrieval, reranking, iterative retrieval, and agentic search with tool calling. Open problems highlighted include evaluation trustworthiness, latency and cost constraints, hallucination and safety, and cross-lingual and multimodal generalization. The entry also provides an engineering checklist covering data privacy, latency budgets, quality validation, security, and serving costs for practitioners moving from paper benchmarks to production systems.

SIGIR 2024 First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval

Overview

This entry catalogs the First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval, held at SIGIR 2024. The official program and accepted papers are listed at: https://llm4eval.github.io/SIGIR2024/papers/

| Field | Content | |-------|---------| | Type | Conference workshop | | Venue | SIGIR 2024 | | Link | https://llm4eval.github.io/SIGIR2024/papers/ | Category | Conferences, Workshops |

Background and Scope

Information retrieval systems at scale have long faced challenges in efficiency, scalability, and user intent understanding. Traditional pipeline approaches separate retrieval, ranking, and generation, which struggles to meet modern demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. This workshop was organized to systematically advance the theory and practice of using LLMs for evaluation in this intersection area.

Core scenarios addressed by the workshop papers include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation, and end-to-end architectures combining external knowledge sources with generative models.

Key Themes

  • A unified perspective on LLM-based evaluation, bringing scattered work into a comparable framework.
  • Clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
  • Reproducible benchmarks, datasets, and taxonomies that lower the entry barrier for subsequent researchers.
  • Interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, with migration paths from research prototypes to industrial systems.
  • Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal extension.
  • Typical Method Pipeline

    1. Input and representation: encode queries, documents, and user context into dense/sparse representations or structured prompts. 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, chained or run in parallel per task. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrap data synthesis. 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.

    Insights for Search, Recommendation, and Personalization

    1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm is making "retrieval count and policy" itself a learnable object. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human assessment. 4. Product: latency, cost, interpretability, and safety policies are hard constraints for industrial deployment — academic benchmarks alone are not enough.

    Engineering Checklist

    | Item | Question | Suggestion | |------|----------|------------| | Data | Does training/index data contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stop, cache popular queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Small-model routing, distillation, hybrid sparse+dense retrieval |

    Limitations

    Possible limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, unknown cross-lingual generalization from English-centric data, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.

    Glossary

    | Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |

    Related Entries

  • 2025 SIGIR Workshop on eCommerce
  • CIKM 2024 1st Workshop on Multimodal Search and Recommendations
  • EACL 2024 Workshop on Personalization of Generative AI Systems
  • ICDM MMSR 2025
  • References

  • Original: SIGIR 2024 First Workshop on LLMs for Evaluation in Information Retrieval. See workshop paper listing.

Tags

#sigir-2024#llm-evaluation#information-retrieval#llm-as-judge#rag#search#benchmarks#workshop

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208510