LLMRec: Large Language Models with Graph Augmentation for Recommendation (WSDM 2024)

LLMRec: Large Language Models with Graph Augmentation for Recommendation

> Published at WSDM 2024. DOI: 10.1145/3616855.3635853 > Link: https://dl.acm.org/doi/abs/10.1145/3616855.3635853

Key points

• Problem: Collaborative filtering recommender systems suffer from data sparsity and cold-start items, which limit the quality of learned user–item representations, especially for graph-based models such as LightGCN.
• Core idea: Use a large language model as an external knowledge source to augment the user–item interaction graph, then distill the LLM-derived signal into the recommender through structure learning, without serving the LLM online.
• Method components:
• Prompt an LLM to produce auxiliary item descriptions, related items, and inferred attributes.
• Construct an augmented graph that combines the original interaction edges with LLM-derived edges / features.
• Train a graph-based recommender to align its structure-aware representations with LLM-augmented signals via a structure learning objective.
• Empirical findings:
• Consistent improvements over strong baselines on public benchmarks.
• Gains are largest for sparse users and cold-start items, indicating that LLM-derived knowledge transfers useful semantic priors into the graph.
• Augmentation is performed offline, so the recommendation model retains the inference cost of a standard GNN at serving time.
• Open issues identified:
• Risk of LLM hallucination propagating into recommendation graphs.
• Cost and latency of offline LLM augmentation at industrial scale.
• Fairness and bias amplification when LLM priors favor already-popular items.
• Gap between synthetic augmentation quality and online user behavior distributions.

Method overview

1. Input: User–item interaction graph G = (V, E) with users U and items I. 2. LLM augmentation: For each item (or a sampled subset), query an LLM with a structured prompt to generate textual descriptions, semantically related items, and inferred attributes. These outputs are converted into additional nodes or edges G' = (V, E ∪ E_llm). 3. Structure learning: A graph-based recommender (e.g., LightGCN) is trained on the augmented graph with a joint objective that preserves collaborative signals while aligning representations with LLM-derived structure. 4. Inference: Only the GNN recommender is deployed; the LLM is not required at serving time.

Experimental setup (as reported)

• Datasets: Public recommendation benchmarks commonly used in graph-based CF literature.
• Baselines: Classical matrix factorization, sequential models, and graph-based recommenders such as LightGCN.
• Metrics: Recall@k, NDCG@k, and Hit Rate.
• Ablations: Effect of removing each LLM-derived signal (text, related items, attributes); impact of augmentation ratio.

Exact numerical results should be verified against the published PDF before citation.

Implications for the field

• Architecture: Offline LLM augmentation is a practical compromise — the LLM improves training data quality, while a lightweight GNN keeps serving latency low.
• Data: Augmentation pipelines must guard against knowledge leakage and distribution shift between LLM priors and real user behavior.
• Evaluation: Offline Recall/NDCG gains should be confirmed with online A/B tests; hallucinated edges can inflate offline metrics without improving real CTR.
• Product: Latency, cost, and explainability constraints remain the dominant blockers for industrial adoption of LLM-in-the-loop recommenders.

Limitations and future work

• Limited GPU budgets may restrict how many items are augmented with LLM calls.
• Benchmarks may not reflect real user distributions, especially under heavy long-tail behavior.
• Cross-lingual and cross-domain generalization of LLM-augmented graphs is still open.
• Future directions include test-time compute allocation for adaptive augmentation, deeper integration with knowledge graphs and structured item databases, and explicit causal and fairness constraints on augmented edges.

Cross-references

• 360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation
• Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Recommendation
• Augmenting Netflix Search with In-Session Adapted Recommendations, RecSys 2022
• Bridging Language and Items for Retrieval and Recommendation (Mar 2024)
• Data-efficient Fine-tuning for LLM-based Recommendation, SIGIR 2024
• DiffKG: Knowledge Graph Diffusion Model for Recommendation, WSDM 2024

Engineering checklist

| Check | Question | Suggestion | |-------|----------|------------| | Data | Does augmentation contain PII or stale item info? | Versioned indices, scrub PII, roll back embeddings | | Latency | What is the p99 budget, and how many items need LLM calls? | Sample items, batch prompts, cache outputs | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving, human audits, citation checks | | Safety | Could LLM hallucinations inject poisoned edges? | Source gating, adversarial detection, output filtering | | Cost | Token and GPU cost per augmented item? | Small-model routing, distillation, periodic refresh |

Glossary

| Term | Meaning | |------|--------| | CF | Collaborative Filtering | | GNN | Graph Neural Network | | LLM | Large Language Model | | RAG | Retrieval-Augmented Generation | | LightGCN | A lightweight graph CF model | | nDCG | Normalized Discounted Cumulative Gain |

Recommendations for readers

• Researchers: Reproduce the core experiments and check whether statistical significance and compute cost are reported.
• Engineers: Extract the augmentation module as a plug-in and evaluate integration cost with an existing CF stack.
• Product managers: Focus on user-perceived gains (relevance, diversity, latency) rather than only offline Recall/NDCG.

Reference

LLMRec: Large Language Models with Graph Augmentation for Recommendation, WSDM 2024. https://dl.acm.org/doi/abs/10.1145/3616855.3635853