Can Large Language Models Understand Preferences in Personalized Recommendation? A PerRecBench Evaluation

Source

| Field | Value | |---|---| | Title | Can Large Language Models Understand Preferences in Personalized Recommendation? | | Authors | Zhaoxuan Tan, Zinan Zeng, Qingkai Zeng, Zhenyu Wu, Zheyuan Liu, Fengran Mo, et al. (7 authors) | | Date | 2025-01-23 | | Link | https://arxiv.org/abs/2501.13391v1 | | Code & Data | https://github.com/TamSiuhin/PerRecBench | | Category | Personalization / Recommender Systems |

Background and Motivation

Large language models (LLMs) have shown strong performance across many tasks, including personalized recommendation. Yet current evaluation protocols typically rely on rating prediction, measuring the regression error between predicted and actual ratings. The authors observe that raw ratings conflate two latent factors: user rating bias (some users systematically rate higher or lower) and item quality (some items are broadly liked or disliked). When both factors are removed by grouping, an LLM that appeared accurate in predicting ratings may fail to identify which items a specific user genuinely prefers.

This observation motivates a rethinking of how LLM-based recommenders should be evaluated, separating the measurement of "preference understanding" from rating regression.

PerRecBench

PerRecBench is the paper's central contribution. It evaluates recommendation techniques by their ability to capture personal preferences in a grouped ranking manner, explicitly disassociating evaluation from user rating bias and item quality. Compared with conventional regression metrics, PerRecBench shows low correlation, indicating that it captures a different—and arguably more user-centric—signal.

Key Findings (19 LLMs Evaluated)

• Size helps, but not enough. Larger LLMs generally outperform smaller ones, yet all evaluated models still struggle with personalized recommendation.
• Ranking format matters. Pairwise and listwise ranking approaches consistently beat pointwise ranking for surfacing users' favored and disfavored items.
• User profiles are important. Incorporating user profiles substantially improves preference recovery, highlighting the value of personalized context.
• Pretraining data matters. The distribution of pretraining data influences how well an LLM understands preferences in a target domain.
• A known-good LLM regressor can still fail at preference ranking. Techniques that excel at rating prediction do not necessarily identify what a user truly likes or dislikes once bias and item quality are controlled.

Supervised Fine-Tuning Strategies

The authors explore three supervised fine-tuning (SFT) strategies aimed at improving preference understanding. The most promising direction is merging model weights trained on single formats, although improving LLMs' understanding of user preferences remains an open research problem.

Why It Matters

• For researchers: PerRecBench offers a bias-controlled, ranking-based alternative to rating regression, making it easier to isolate "true" preference signals from confounders.
• For practitioners: The findings warn against assuming that strong rating-prediction numbers translate into accurate personalization. Pairwise/listwise formulations, richer user profiles, and careful SFT (including weight merging) are more promising levers.

Limitations and Open Problems

• Even the best LLMs evaluated still struggle with personalized recommendation.
• Improvements from SFT are uneven, and it remains unclear how to robustly teach LLMs preference semantics.
• The role of pretraining corpora is acknowledged but not fully characterized.
• Generalization across domains, languages, and user populations is left for future work.

Links

• Paper: https://arxiv.org/abs/2501.13391v1
• Code & data: https://github.com/TamSiuhin/PerRecBench

Original Abstract

> Large Language Models (LLMs) excel in various tasks, including personalized recommendations. Existing evaluation methods often focus on rating prediction, relying on regression errors between actual and predicted ratings. However, user rating bias and item quality, two influential factors behind rating scores, can obscure personal preferences in user-item pair data. To address this, we introduce PerRecBench, disassociating the evaluation from these two factors and assessing recommendation techniques on capturing the personal preferences in a grouped ranking manner. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated by grouping users. With PerRecBench and 19 LLMs, we find that while larger models generally outperform smaller ones, they still struggle with personalized recommendation. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions. We further explore three supervised fine-tuning strategies, finding that merging weights from single-format training is promising but improving LLMs' understanding of user preferences remains an open research problem. Code and data are available at https://github.com/TamSiuhin/PerRecBench