Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey
Source: https://arxiv.org/abs/2404.00621 · Type: Survey · Posted: 2024-03-31 · Authors: Qijiong Liu, Jieming Zhu, Yanting Yang, Quanyu Dai, Zhaocheng Du, Xiao-Ming Wu, et al. (9 authors)
Overview
Personalized recommendation is a ubiquitous channel for users to discover content tailored to their interests. However, traditional recommendation models primarily rely on unique IDs and categorical features for user-item matching, overlooking the nuanced essence of raw item contents across modalities such as text, image, audio, and video. This underutilization is especially limiting for multimedia services like news, music, and short-video platforms. Recent advances in large multimodal models create new opportunities—and challenges—for building content-aware recommender systems. This survey provides a comprehensive exploration of multimodal pretraining, adaptation, and generation techniques and their applications to recommendation.
Key points
- Unified perspective: Consolidates scattered work on multimodal recommendation into a comparable framework spanning representation learning, retrieval, reranking, planning, generation, and feedback mechanisms.
- Motivation: ID-based matching misses semantic signals in raw content; multimodal models offer semantic priors and cold-start capabilities, at the cost of inference expense and hallucination risk.
- Dense retrieval: high recall, low latency; suited to first-stage retrieval.
- Late interaction (e.g., ColBERT): higher precision but larger indexes.
- Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
- Agentic search: frames search as sequential decision-making, enabling multi-hop reasoning and self-reflection.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
Taxonomy
| Dimension | Subcategories | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Two-tower, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B | Comparable | Deviates from real satisfaction |
Four research threads
Timeline
2019–2021: BERT rerankers and DPR established neural retrieval; 2022–2023: RAG and FreshLLM fused retrieval with generation; 2024 onward: conversational/agentic search and Gen-RecSys exploded; 2025–2026: RL-trained search agents, Deep Research, and GraphRAG are emerging growth areas.
Evaluation landscape
Takeaways
1. Architecture: cascaded retrieve–rerank–generate remains mainstream, but agentic paradigms make retrieval strategy itself learnable. 2. Data: high-quality instruction data and click/session logs are critical; synthetic data risks leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Production: latency, cost, explainability, and safety are hard constraints for deployment—do not optimize academic benchmarks alone.
Open challenges
Insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future directions include finer-grained process supervision, joint retrieval-reasoning training, multimodal and cross-lingual consistency, and causal/fairness constraints for recommenders.
> Original abstract (excerpt): "Personalized recommendation serves as a ubiquitous channel for users to discover information tailored to their interests... This survey seeks to provide a comprehensive exploration of the latest advancements and future trajectories in multimodal pretraining, adaptation, and generation techniques, as well as their applications in enhancing recommender systems."