Revisiting Prompt Engineering for LLM-Based Personalized Recommendation: A Comprehensive Evaluation

Introduction

With the rise of large language models (LLMs), leveraging natural-language prompts to perform recommendation tasks has become feasible. Compared with traditional collaborative-filtering methods, LLM-driven recommendation offers distinct advantages in cold-start, cross-domain, and zero-shot scenarios, while supporting flexible inputs and generating explanations for user behavior. However, how to design prompts—i.e., prompt engineering—to fully unlock LLMs' potential in recommendation lacks systematic investigation.

Kusano et al.'s paper *"Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation"* addresses this gap through large-scale evaluation. The study focuses on single-user personalized recommendation, using only the target user's own history without other users' data. This setting matters for privacy-sensitive and data-limited applications where prompt engineering becomes the critical lever for output quality.

The authors compare 23 prompt types across 8 public datasets and 12 LLM models, evaluating accuracy and inference cost via statistical tests and linear mixed-effects models—an empirical scale far exceeding prior work.

Advantages and Challenges of LLMs in Personalized Recommendation

LLMs offer multiple benefits for recommendation:

• Cold-start handling: New users or items lacking history can still receive recommendations via the LLM's prior knowledge.
• Cross-domain recommendation: Generic semantic understanding enables transferring patterns across domains.
• Flexible natural-language inputs: Incorporating unstructured descriptions, reviews, and contextual cues is straightforward.
• Explanation generation: LLMs can rationalize why a particular item (e.g., a movie) is recommended, boosting trust.

In the single-user setting, however, the LLM must infer preferences solely from limited personal history, making prompt design especially critical. Differences between cost-efficient and high-performance models mean their responses to the same prompt can vary dramatically, raising practical questions about balancing accuracy and inference cost.

Large-Scale Experimental Design

The study covers 23 prompt types, 8 real-world datasets, and 12 LLM models, with the following key elements:

• Prompt Types: Templates range from simple instructions to chain-of-thought-style variants, spanning standardized phrases, non-conversational prompts, and conversational prompts. Some rephrase instructions, others inject background knowledge (item attributes, user profiles), and others guide stepwise reasoning.
• Datasets: 8 public datasets covering movies, products, and music, with varying sparsity and behavioral patterns to test prompt generalizability.
• LLM Models: 12 models split into cost-efficient (small, fast, cheaper) and high-performance (large, accurate, expensive) categories, enabling cross-capability analysis.
• Metrics: Ranking-quality measures such as nDCG for accuracy, plus inference cost over a fixed workload (e.g., 1,600 users).
• Analysis Methods: Statistical tests validate that accuracy differences are not random; linear mixed-effects models quantify prompt effects while controlling for dataset and model variance.

Key Findings

Cost-Efficient LLMs Benefit from Engineered Prompts

Three prompt strategies proved especially effective for smaller LLMs:

1. Rephrased instructions: Clarifying wording (e.g., expanding "recommend movies" into "recommend movies the user might enjoy based on their preferences") reduces ambiguity. 2. Background knowledge injection: Adding item attributes or preference summaries supplies extra cues that compensate for limited model knowledge. 3. Simplified reasoning: Explicit, step-light reasoning paths prevent small models from wandering into incoherent chains.

High-Performance LLMs Prefer Simple Prompts

Counter-intuitively, for large LLMs, simple prompts often outperform complex ones. Verbose or highly structured instructions can reduce accuracy and increase cost, likely because strong models already possess the internal capability to interpret tasks effectively, and over-guidance constrains their flexibility. Simple prompts preserve high accuracy while lowering compute.

Limits of Common NLP Prompting Techniques

Several widely used NLP techniques did not transfer:

• Step-by-step reasoning prompts, which boost QA and summarization, decreased accuracy in recommendation.
• Reasoning-specialized models (chain-of-thought fine-tuned) also underperformed.

Recommendation appears to rely more on intuitive user-item matching than on strict logical deduction, so over-emphasizing explicit steps can interfere with the model's judgment.

Accuracy vs. Cost Trade-off

Although higher accuracy often requires additional compute, this is not always the case: high-performance models with simple prompts achieve top accuracy at lower cost. For cost-sensitive deployments, the study identifies prompt choices offering the best accuracy-per-dollar ratio.

Practical Guidelines

1. Match prompt complexity to model capability

• Cost-efficient LLMs: Use engineered prompts—rephrased instructions, background knowledge, and simplified reasoning.
• High-performance LLMs: Prefer concise prompts; complex instructions add cost without proportional gains.

2. Balance accuracy and cost explicitly

• For maximum accuracy, pair high-performance models with well-designed prompts.
• For latency- or budget-sensitive services, choose cost-efficient models with proven prompts, or high-performance models with simple prompts.

3. Avoid blindly copying NLP prompting tricks

• Recommendation has unique characteristics centered on user-item matching; prompts should highlight user preferences and item attributes rather than emphasizing logical decomposition.

4. Treat prompt engineering as continuous iteration

• As LLMs and recommendation scenarios evolve, regularly A/B test prompt variants on target user populations and incorporate new findings.

Conclusion

The paper delivers the most comprehensive empirical analysis to date of prompt engineering for single-user LLM-based recommendation. Its core message is that prompt design should be tailored to model capability: small models gain from rich, structured prompts, while large models thrive on simplicity. Critically, common NLP prompting heuristics—step-by-step reasoning, reasoning-specialized models—do not transfer and can degrade performance. These findings provide concrete, evidence-based guidance for practitioners navigating the accuracy-cost trade-off, and they open avenues for future work on automated prompt selection and multimodal prompt design.

References

• Kusano et al., *Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation*.
• Related: arXiv preprint 2024 (search title for DOI/arXiv ID).