Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyond
Source: arXiv:2410.19744 · Published 2024-10-10 · Type: Survey Authors: Qi Wang, Jindong Li, Shiqi Wang, Qianli Xing, Runliang Niu, He Kong, et al. (10 authors in total) Resource list: https://github.com/jindongli-Ai/Next-Generation-LLM-based-Recommender-Systems-Survey
Background and Motivation
Large language models (LLMs) have not only revolutionized NLP but also have the potential to bring a paradigm shift to many other fields due to their remarkable language understanding, generalization, and reasoning abilities. Recent studies have actively attempted to harness LLMs to improve recommender systems, making a thorough review of recent advances and challenges imperative. Traditional pipelines that separately handle retrieval, ranking, and generation struggle to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and real-time knowledge.
Unlike existing surveys that classify LLM-based recommenders mainly by the technical framework of LLMs, this work investigates how LLMs can better serve recommendation tasks from the perspective of the recommender system community, and explicitly addresses the long-standing gap between academic research and industrial application.
Core Contributions
- A unified perspective that organizes scattered related work into a comparable framework.
- A clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) for engineering practice.
- Reproducible benchmarks, datasets, and classification tables that lower the entry barrier for follow-up research.
- Discussion of interfaces with LLM tool calling, reinforcement learning, and multi-agent collaboration, and paths from research prototypes to industrial systems.
- Explicit open problems: evaluation credibility, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of retrieval steps, reranking depth, training data scale.
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain
- Agentic Search: Modeling search as sequential decisions and tool calls
- Gen-IR: Generative Information Retrieval
- A Comprehensive Survey on Retrieval Methods in Recommender Systems (arXiv:2407.21022)
- A Survey of Generative Search and Recommendation in the Era of Large Language Models (arXiv:2404.16924)
- A Survey on LLM-powered Agents for Recommender Systems (arXiv:2502.10050)
- A Comprehensive Survey on Cross-Domain Recommendation (arXiv:2503.14110)
- A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys, ACM 10.1145/3637528.3671474)
The Proposed Three-Tier Taxonomy
The survey introduces a novel taxonomy rooted in the intrinsic essence of recommendation, structured in three tiers reflecting progression from research to practical implementation:
1. Representing and understanding — how LLMs encode users, items, and interactions; semantic priors and cold-start capability. 2. Scheming and utilizing — how LLMs plan and act: retrieval-augmented generation (RAG), agent/tool-use paradigms, reasoning over user intent. 3. Industrial deployment — engineering constraints for real-world systems.
Method Landscape
| Dimension | Sub-type | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Two-tower, cross-encoder, DSI | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Divergence from real satisfaction |
The field has evolved through several lines: dense retrieval (high recall, low latency, suited to first-stage retrieval), late interaction (e.g., ColBERT — higher precision but larger indexes), generative IR (generating documents or docids directly, simplifying cascades), and agentic search (modeling search as sequential decision-making with multi-hop and self-reflection).
Timeline: 2019–2021, BERT rerankers and DPR established neural retrieval; 2022–2023, RAG and FreshLLM drove retrieval-generation fusion; from 2024, conversational/agentic search and Gen-RecSys surged; 2025–2026, RL-trained search agents, Deep Research, and GraphRAG are emerging growth areas.
Evaluation Paradigm
Quantitative results should be verified against the original PDF tables.
Key Insights
1. Architecture: Cascaded retrieve–rerank–generate remains mainstream, but agentic paradigms make retrieval count and strategy themselves learnable. 2. Data: High-quality instruction data and click/session logs are equally critical; synthetic data risks knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product: Latency, cost, explainability, and safety are hard constraints for industrial deployment—not just academic benchmarks.
Open Problems and Future Directions
The authors note insufficient unified benchmarks, irreproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal/cross-lingual consistency. Additional limitations include GPU-budget-bound experiments, benchmarks misaligned with real user distributions, English-centric data, and security risks of agents on the open web.
Engineering Checklist (from the community digest)
| Item | Question | Recommendation | |---|---|---| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stopping, hot-query caching, async reranking | | Quality | Does offline gain transfer to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias from open retrieval? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |