Pre-train, Prompt, and Recommendation: A Comprehensive Survey of Language Modeling Paradigm Adaptations in Recommender Systems (TACL, Dec 2023)
- Source: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00619/118712
- Type: Survey
- Section: Recommender Engine Surveys
- The survey systematically reviews how the pre-train and prompt paradigms from language modeling have been adapted for recommender systems.
- It offers a unified view that organizes scattered prior work into a comparable framework, decomposing method components such as representation learning, retrievers, re-rankers, planners, generators, and feedback mechanisms.
- It discusses interfaces with emerging paradigms including LLM tool calling, reinforcement learning, and multi-agent collaboration.
- 2019–2021: BERT re-ranking and DPR establish neural retrieval foundations.
- 2022–2023: RAG and retrieval–generation fusion gain momentum; generative recommendation (Gen-RecSys) emerges.
- 2024+: Conversational/agentic search, RL-trained search agents, Deep Research, and GraphRAG become new growth areas.
- A Comprehensive Survey on Retrieval Methods in Recommender Systems
- A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)
- A Survey of Generative Search and Recommendation in the Era of Large Language Models
- A Survey on LLM-powered Agents for Recommender Systems
Key points
Taxonomy covered
| Dimension | Sub-categories | Strengths | Limitations | |-----------|----------------|-----------|-------------| | Modeling paradigm | Discriminative vs. generative retrieval (dual-tower, cross-encoders, DSI) | Mature, scalable | Semantic drift, index update cost | | LLM integration | RAG / Agents / Tool use | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness (multi-objective LTR, RLHF) | Aligned with business goals | Annotation scarcity | | Evaluation | Offline / online / human (nDCG, MRR, LLM-as-judge, A/B) | Comparable | Divergence from real satisfaction |
Evolution of the field
Insights for search/recommendation systems
1. Architecture: cascade retrieval + re-ranking + generation remains mainstream, but agentic paradigms make retrieval strategy itself learnable. 2. Data: high-quality instruction data and click/session logs are critical; synthetic data risks knowledge leakage and distribution shift. 3. Evaluation: gaps between offline metrics and online satisfaction are widening; LLM-as-judge needs cross-validation with human assessment. 4. Deployment: latency, cost, interpretability, and safety are hard industrial constraints.
Open problems
Lack of unified benchmarks, irreproducibility on private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future directions include fine-grained process supervision, joint retrieval-reasoning training, multimodal and cross-lingual consistency, and causal/fairness constraints for recommendation.