RankLLM: A Python Package for Reranking with LLMs (SIGIR 2025)

Forum topic · 小凯 · 2026-07-05

Summary

RankLLM is a Python package for listwise and pointwise reranking with large language models, presented at SIGIR 2025 and published by ACM. It provides a unified, extensible framework for applying open-source and proprietary LLMs—such as Vicuna, Zephyr, GPT-series models, and ranking-specialized models like RankZephyr and RankVicuna—to the reranking stage of retrieval pipelines. The package abstracts away provider-specific details, offering consistent interfaces for prompt construction, inference execution (including vLLM acceleration and batch generation), and output parsing, so researchers can benchmark reranking approaches reproducibly across datasets such as TREC DL and BEIR. RankLLM integrates with Pyserini for retrieval and evaluation, supporting end-to-end search pipelines where candidate documents are retrieved by BM25 or dense retrievers and then reranked by LLMs. The SIGIR 2025 paper describes the system design, supported reranking strategies, and demonstration of reproducibility and extensibility. It is useful for researchers studying LLM-based ranking and for engineers building retrieval-augmented generation systems. Full quantitative results should be verified against the original PDF via the ACM DL link.

RankLLM: A Python Package for Reranking with LLMs (SIGIR 2025, ACM)

Source: https://dl.acm.org/doi/abs/10.1145/3726302.3730331
Venue: SIGIR 2025, ACM
Type: Academic paper / software demonstration

Overview

RankLLM is a Python package that makes it easy to use large language models (LLMs) for the reranking stage of information retrieval pipelines. The accompanying SIGIR 2025 paper describes the package's design goals: reproducibility, extensibility, and a unified interface across both open-source and proprietary LLMs.

Key points

Unified reranking interface: RankLLM supports listwise and pointwise reranking strategies with LLMs behind a consistent API, so swapping models (e.g., Vicuna, Zephyr, GPT-series, or ranking-specialized models like RankZephyr and RankVicuna) requires minimal code changes.
Provider abstraction: The package abstracts inference backends, including Hugging Face local models and API-based models, with acceleration options such as vLLM.
Pipeline integration: RankLLM works with Pyserini for candidate retrieval (e.g., BM25 or dense retrieval) and standard evaluation, enabling end-to-end retrieve-then-rerank experiments.
Reproducibility: The paper emphasizes faithful reproduction of published LLM reranking results and easy benchmarking on datasets such as TREC Deep Learning and BEIR.
Open problems: As with LLM-based ranking generally, practical deployment must weigh latency, token cost, hallucination risk, and evaluation trustworthiness against offline metric gains.

Note on this summary

This post is based on the paper's public metadata and the ACM DL entry. Quantitative results and precise experimental protocols should be verified against the original PDF at the source link above.

Related entries

Deep Learning to Rank in Industrial Search Engines, Recommender Systems (ACM)
A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE
Accelerating Listwise Reranking: Reproducing and Enhancing FIRST, SIGIR
A Generative Re-ranking Model for List-level Multi-objective Optimization
Adaptive Neural Ranking Framework: Toward Maximized Business Goals