Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models (RecSys 2024)
Overview
This post indexes the RecSys 2024 paper "Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models", filed under the Recommender Engines section of a curated reading list.
- Source: https://dl.acm.org/doi/abs/10.1145/3640457.3688104
- Venue: RecSys 2024
- Type: Academic paper
- A unified perspective for the problem domain, bringing scattered related work into a comparable framework.
- A clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) to ease engineering adoption.
- Reproducible benchmarks, datasets, or taxonomies, lowering entry barriers for follow-up research.
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, pointing toward paths from research prototypes to industrial systems.
- Explicit open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets;
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost;
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs;
- Ablations: contribution of retrieval steps, rerank depth, and training data scale.
- 360Brew: A Decoder-only Foundation Model for Personalized Ranking
- Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers
- Augmenting Netflix Search with In-Session Adapted Recommendations (RecSys)
- Bridging Language and Items for Retrieval and Recommendation
- Data-efficient Fine-tuning for LLM-based Recommendation (SIGIR 2024)
- DiffKG: Knowledge Graph Diffusion Model for Recommendation (WSDM 2024)
- Original paper: *Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models*, RecSys 2024. https://dl.acm.org/doi/abs/10.1145/3640457.3688104
One-line takeaway
The work examines how knowledge augmentation from large language models (LLMs) can push recommender systems toward open-world settings.
Background and Motivation
In large-scale search, recommendation, and personalization systems, recommendation has long faced challenges in efficiency, scalability, and user-intent understanding. Traditional pipelined approaches often treat retrieval, ranking, and generation as disjoint stages, making it hard to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and real-time knowledge. This paper is proposed against that backdrop, aiming to advance the theoretical and practical boundaries of this intersection.
Core problem settings include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation, and end-to-end architectures that couple external knowledge sources with generative models.
Core Contributions
Typical Method / System Architecture
Work in this area generally follows: problem formalization → model/system design → training or construction pipeline → inference pipeline:
1. Input and representation: encode queries, documents, and user context as dense/sparse representations or structured prompts; 2. Core modules: retrievers, rerankers, planners, memory modules, tool interfaces, chained or branched by task; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, RL (including process rewards), bootstrapped data synthesis; 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Evaluation
Typical evaluation setups in this space include:
Specific numbers should be verified against the original PDF; this report is based on the abstract and public metadata.
Insights for Search / Rec / Personalization
1. Architecture: cascaded retrieval + rerank + generation remains mainstream, but the agentic paradigm is making "retrieval count and policy" itself learnable; 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation; 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment—do not optimize academic benchmarks alone.
Limitations and Future Work
Likely limitations include experiment scale constrained by GPU budget, mismatch between benchmarks and real user distributions, English-centric data leaving cross-lingual generalization unknown, and safety risks of agents operating on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.
Cross-References
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |