SIGIR 2024: The Second Workshop on Generative Information Retrieval (Gen-IR '24)

Overview

| Field | Detail | |-------|--------| | Title | SIGIR 2024 The Second Workshop on Generative Information Retrieval | | Venue | Workshop co-located with SIGIR 2024 | | Link | Workshop page | | Resource type | Conference / Workshop | | Category | Conferences, Workshops |

Background and Scope

Information retrieval in large-scale search, recommendation, and personalization systems has long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipelined approaches tend to separate retrieval, ranking, and generation, which struggles to meet the LLM-era demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. The Second Workshop on Generative Information Retrieval (Gen-IR '24) at SIGIR 2024 was organized to systematically advance the theory and practice of this intersection.

Core scenarios addressed include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommender systems, and end-to-end architectures that coordinate external knowledge sources with generative models.

Core Topics and Contributions

• A unified perspective that brings together scattered related work into a comparable framework.
• A clear decomposition of method components (representation learning, retrievers, re-rankers, planners, generators, feedback mechanisms) to support engineering practice.
• Reproducible benchmarks, datasets, and taxonomies from experimental protocols and survey coverage, lowering the barrier for follow-up researchers.
• Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, with paths from research prototypes to industrial systems.
• Explicit open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.

Typical Methodological Pipeline

Methods in this space usually follow four steps: problem formalization → model/system design → training or construction → inference pipeline.

1. Input and representation: encode queries, documents, and user context as dense or sparse representations, or build structured prompts; 2. Core modules: retrievers, re-rankers, planners, memory modules, tool interfaces — chained or parallel by task; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), bootstrapped data synthesis; 4. Inference strategies: single-round retrieval, iterative retrieval, parallel sub-queries, early stopping and budget control.

Position within the SIGIR / RecSys Ecosystem

See the official workshop website for details on organizers, accepted papers, and schedule.

Key Takeaways for Search / Rec / Personalization

1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but the agentic paradigm increasingly treats "when and how often to retrieve" as a learnable decision; 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human assessment; 4. Product: latency, cost, interpretability, and safety policies are hard constraints for industrial deployment — academic benchmarks alone are insufficient.

Engineering Checklist

| Item | Question | Suggestion | |------|----------|------------| | Data | Does training/indexing contain PII? How are versions managed? | Partitioned indices, sanitization, rollback-able embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade + early stop, cache hot queries, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Route to smaller models, distillation, hybrid sparse+dense |

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |

Limitations and Notes

Likely limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric data with unknown cross-lingual generalization, and safety risks of agentic systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommender systems.

Related Entries

• 2025 SIGIR Workshop on eCommerce
• CIKM 2024 1st Workshop on Multimodal Search and Recommendations
• EACL 2024 Workshop on Personalization of Generative AI Systems
• ICDM MMSR 2025

References

• Original entry: SIGIR 2024 The Second Workshop on Generative Information Retrieval. See the linked workshop page for publication details.

Deeper Analysis Appendix

Technical Lineage

This work sits at the intersection of information retrieval and large-scale search/recommendation. From a systems perspective, it addresses the core question of "how to redistribute responsibility among retrieval, ranking, generation, and tool calling in the LLM era." If the classic search stack is a funnel — recall for coverage, fine ranking for discrimination, generation for presentation — the new variables in the LLM era are the reasoning budget and action space (whether to retrieve, how many times, and which tools to invoke).

Related Work Landscape

Neural IR has evolved from BM25 to BERT cross-encoders, dual-tower dense retrieval, late interaction, and on to generative retrieval and LLM agents. Each generation balances the triangle of efficiency–effectiveness–maintainability. Dense retrieval achieves millisecond recall via approximate nearest-neighbor search but is sensitive to domain shift and long-tail queries; cross-encoders are accurate but cannot precompute document representations; generative methods reduce cascade errors but face index-update difficulties.

On the recommendation side, the progression from matrix factorization through deep CTR models, sequential Transformers, and LLM instruction-following / generative recommendation (Gen-Rec) faces a persistent tension: sparse user behavior, enormous item catalogs, and multi-objective business trade-offs. LLMs provide semantic priors and cold-start capability, but online inference cost and hallucination risk demand careful system design.

RAG and Agentic Search extend external knowledge access from a single retrieval into an iterative, verifiable, plannable process; evaluation correspondingly shifts from static nDCG toward task success rate, citation accuracy, and multi-hop reasoning chain completeness.

From Abstract to Implementation

Paper abstracts emphasize headline metrics, but engineering teams must additionally evaluate index update frequency, embedding version compatibility, online A/B sensitivity, and failure modes (empty retrieval, wrong tool calls, over-generation). Readers should pair this entry with related surveys, open-source implementations (e.g., RankLLM, Open Deep Research), and industrial blogs to triangulate "paper innovation — open-source reproduction — production constraints."

Actionable Advice

1. Researchers: reproduce core comparisons; check whether statistical significance and compute cost are reported; 2. Engineers: extract pluggable modules (encoders, re-rankers, planners) and assess integration cost with existing stacks; 3. Product managers: identify user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.