LLMRank: Large Language Models as Zero-Shot Rankers for Recommender Systems (ECIR 2024, Springer)

Forum topic · 小凯 · 2026-07-05

Summary

This ECIR 2024 paper (Springer, LNCS chapter) introduces LLMRank, showing that large language models can serve as zero-shot rankers for recommender systems without any task-specific training. The authors frame recommendation ranking as a conditional ranking task over a candidate item set and prompt an LLM to order candidates using its learned world knowledge. Two key techniques are proposed: (1) natural language prompting that converts user behavior sequences into a readable description of user preference, and (2) bootstrapping, in which few-shot in-context examples are iteratively refined using the model's own high-confidence predictions and the target recommender's feedback. Experiments on public benchmarks show that an off-the-shelf LLM with LLMRank achieves ranking performance comparable to or better than strongly supervised baseline rankers in the zero-shot setting. The work demonstrates that LLMs encode transferable preference signals useful for recommendation, offering a low-cost alternative to supervised ranking models and a pathway for cold-start scenarios. Full details: https://link.springer.com/chapter/10.1007/978-3-031-56060-6_24

LLMRank: Large Language Models are Zero-Shot Rankers for Recommender Systems (ECIR 2024, Springer)

Overview

Paper: Large Language Models are Zero-Shot Rankers for Recommender Systems (LLMRank)
Venue: ECIR 2024, Springer LNCS (March 2024)
Link: https://link.springer.com/chapter/10.1007/978-3-031-56060-6_24
Area: LLM-based recommendation, zero-shot ranking

Key Idea

The paper investigates whether a large language model (LLM), used without any task-specific training, can act as a ranking model for recommendation. The authors cast ranking as a conditional ranking task: given a user's behavior history and a candidate item set, the LLM is prompted to output a ranked list of items directly.

Method

1. Natural language prompting: The user's interaction sequence is converted into natural language descriptions, letting the LLM infer the user's preference from its embedded world knowledge rather than from co-occurrence statistics alone. 2. Bootstrapping: Few-shot demonstrations for in-context learning are constructed automatically. The system iteratively selects examples from the model's own high-confidence predictions and refines them using the target recommender's feedback, avoiding costly manual annotation.

Findings

An off-the-shelf LLM combined with LLMRank achieves zero-shot ranking performance comparable to or better than strongly supervised ranking models on public recommendation benchmarks.
LLMs encode generalizable preference knowledge that transfers across users and domains, which is valuable for cold-start and low-resource scenarios.
Bootstrapped in-context examples improve ranking quality over naive zero-shot prompting.

Significance

LLMRank provides evidence that LLMs can be plug-and-play rankers in recommendation pipelines, reducing the need for supervised training data. It situates itself in the broader trend of applying LLMs to recommender systems, alongside retrieval-augmented and generative recommendation approaches. Engineering considerations—such as inference latency, token cost, and hallucination risk—remain open challenges for production deployment.

References

Original chapter: Springer, ECIR 2024