RAGtifier: Evaluating RAG Generation Approaches in State-of-the-Art RAG Systems (SIGIR LiveRAG Competition)

Forum topic · 小凯 · 2026-07-05

Summary

RAGtifier is a paper describing the system developed for the SIGIR 2025 LiveRAG Competition, which evaluates different retrieval-augmented generation (RAG) approaches and generation strategies in state-of-the-art RAG systems. The authors — Tim Cofala, Oleh Astappiev, William Xiong, and Hailay Teklehaymanot — describe a system built around a retriever-augmented LLM pipeline that assesses how generation methods affect answer correctness and groundedness. The work situates itself in the broader context of open-domain information access, conversational search, and end-to-end architectures that couple external knowledge sources with large language models. Key themes include decomposing RAG pipelines into retrievers, rerankers, generators, and feedback mechanisms; comparing single-turn versus iterative retrieval strategies; and measuring quality with metrics beyond traditional ranking scores, such as task success and citation accuracy. The paper also discusses practical constraints like latency, cost, hallucination control, and evaluation trustworthiness. Full details and results are available in the original arXiv paper (arXiv:2506.14412).

RAGtifier: Evaluating RAG Generation Approaches in State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

Authors: Tim Cofala, Oleh Astappiev, William Xiong, Hailay Teklehaymanot Source: arXiv:2506.14412

Overview

RAGtifier is the system paper describing the team's entry for the SIGIR 2025 LiveRAG Competition. It evaluates RAG generation approaches within state-of-the-art retrieval-augmented generation systems, examining how different generation strategies affect answer quality and groundedness.

Background

In large-scale search and recommendation systems, RAG has long faced challenges around efficiency, scalability, and user-intent understanding. Traditional pipelines treat retrieval, ranking, and generation as disconnected stages, which struggles to meet LLM-era requirements for natural-language interaction, multi-hop reasoning, and real-time knowledge. RAGtifier was proposed in this context to systematically examine the practical boundaries of RAG generation.

Key themes

Unified perspective on RAG pipelines, decomposing them into retrievers, rerankers, planners, generators, and feedback mechanisms
Generation strategy evaluation: comparing how different generation approaches influence correctness, groundedness, and citation accuracy
Retrieval strategies: single-turn retrieval vs. iterative retrieval, parallel sub-queries, early stopping, and compute-budget control
Emerging paradigms: interfaces to LLM tool calling, reinforcement learning, and multi-agent collaboration
Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension

Evaluation context

RAG systems are typically benchmarked with datasets such as MS MARCO, BEIR, and Natural Questions, using metrics like nDCG@10, MRR, Recall@k, task success rate, latency, and token cost. In live/competitional settings, evaluation increasingly shifts from static ranking metrics toward task success, citation accuracy, and multi-hop reasoning completeness. Exact numerical results should be verified against the original PDF.

Takeaways for practitioners

1. Architecture: cascade retrieve–rerank–generate remains mainstream, but agentic approaches make retrieval frequency and policy themselves learnable 2. Data: high-quality instruction data and session logs matter; synthetic data requires leakage and distribution-shift safeguards 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human assessment 4. Deployment: latency, cost, explainability, and safety are hard constraints in production — not just academic benchmarks

Reference

Original paper: RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition — arXiv:2506.14412