Can Large Language Models Understand Preferences in Personalized Recommendation?
Source: arXiv:2501.13391 | Code & Data: https://github.com/TamSiuhin/PerRecBench | Published: 2025-01-23
Authors: Zhaoxuan Tan, Zinan Zeng, Qingkai Zeng, Zhenyu Wu, Zheyuan Liu, Fengran Mo, et al. (7 authors)
TL;DR
The paper introduces PerRecBench, a benchmark that disentangles personalized recommendation evaluation from user rating bias and item quality. LLMs that excel at rating prediction often fail to identify users' favored and disfavored items once these confounders are removed.
Background and Motivation
LLMs have shown strong performance in personalized recommendation, but existing evaluation methods typically focus on rating prediction, measuring regression error between actual and predicted ratings. The authors point out that two influential factors — user rating bias and item quality — can obscure true personal preferences in user-item interaction data, making traditional metrics unreliable for measuring preference understanding.
PerRecBench
PerRecBench disassociates the evaluation from rating bias and item quality by grouping users, then assesses recommendation techniques on capturing personal preferences in a grouped ranking manner.
Key Findings
- 19 LLMs evaluated: larger models generally outperform smaller ones, but all still struggle with personalized recommendation.
- Ranking strategy matters: pairwise and listwise ranking approaches outperform pointwise ranking.
- Low correlation between PerRecBench scores and traditional regression metrics — good rating prediction does not imply good preference understanding.
- User profiles play an important role in preference capture.
- Pretraining data distributions influence recommendation performance.
- Supervised fine-tuning: three SFT strategies were explored; merging weights from single-format training is promising, but improving LLMs' understanding of user preferences remains an open research problem.
- A Survey of Personalization: From RAG to Agent (arXiv:2504.10147)
- Bridging Personalization and Control in Scientific Personalized Search (arXiv:2411.02790)
- IntentRec: Predicting User Session Intent with Hierarchical Multi-Task Learning (arXiv:2408.05353)
- Unified Embedding Based Personalized Retrieval in Etsy Search (arXiv:2306.04833)
- User-LLM: Efficient LLM Contextualization with User Embeddings (WWW, DOI: 10.1145/3701716.3715463)
Original Abstract (English)
> Large Language Models (LLMs) excel in various tasks, including personalized recommendations. Existing evaluation methods often focus on rating prediction, relying on regression errors between actual and predicted ratings. However, user rating bias and item quality, two influential factors behind rating scores, can obscure personal preferences in user-item pair data. To address this, we introduce PerRecBench, disassociating the evaluation from these two factors and assessing recommendation techniques on capturing the personal preferences in a grouped ranking manner. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated by grouping users. With PerRecBench and 19 LLMs, we find that while larger models generally outperform smaller ones, they still struggle with personalized recommendation. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions. We further explore three supervised fine-tuning strategies, finding that merging weights from single-format training is promising but improving LLMs' understanding of user preferences remains an open research problem.