Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey

Overview

• Title: Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey
• Authors: Qijiong Liu, Jieming Zhu, Yanting Yang, Quanyu Dai, Zhaocheng Du, Xiao-Ming Wu, et al. (9 authors total)
• Date: 2024-03-31
• Source: https://arxiv.org/abs/2404.00621
• Type: Survey

Abstract (original)

> Personalized recommendation serves as a ubiquitous channel for users to discover information tailored to their interests. However, traditional recommendation models primarily rely on unique IDs and categorical features for user-item matching, potentially overlooking the nuanced essence of raw item contents across multiple modalities such as text, image, audio, and video. This underutilization of multimodal data poses a limitation to recommender systems, especially in multimedia services like news, music, and short-video platforms. The recent advancements in large multimodal models offer new opportunities and challenges in developing content-aware recommender systems. This survey seeks to provide a comprehensive exploration of the latest advancements and future trajectories in multimodal pretraining, adaptation, and generation techniques, as well as their applications in enhancing recommender systems. Furthermore, we discuss current open challenges and opportunities for future research in this dynamic domain.

Background and Motivation

Large-scale search and recommendation systems have long faced challenges in efficiency, scalability, and user intent understanding. Traditional pipelines treat retrieval, ranking, and generation as disconnected stages. Traditional recommenders built on ID embeddings and categorical features miss the nuanced semantics of raw item content across text, image, audio, and video—a critical limitation for multimedia platforms. Advances in large multimodal models (LMMs) create new opportunities for content-aware recommendation, which motivates this systematic survey of the intersection.

Key Points

• Unified perspective: Consolidates scattered multimodal recommendation work into a comparable framework spanning pretraining, adaptation, and generation.
• Component decomposition: Clarifies roles of representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms for engineering practice.
• Evolving landscape: Neural retrieval progressed from BM25 → BERT cross-encoders → dense retrieval (DPR) → late interaction (ColBERT) → generative IR → LLM agents; on the recommendation side, from matrix factorization → deep CTR → sequential Transformers → LLM-based generative recommendation (Gen-Rec).
• Timeline: 2019–2021 BERT reranking and DPR; 2022–2023 RAG-style retrieval-generation fusion; 2024+ conversational/agentic search and Gen-RecSys; 2025–2026 RL-trained search agents and GraphRAG.
• LLM integration: Discusses interfaces with RAG, tool use, and multi-agent collaboration, and pathways from research prototypes to industrial systems.

Related Survey Entries

• A Comprehensive Survey on Retrieval Methods in Recommender Systems
• A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)
• A Survey of Generative Search and Recommendation in the Era of Large Language Models
• A Survey on LLM-powered Agents for Recommender Systems
• A Comprehensive Survey on Cross-Domain Recommendation

Open Challenges

• Insufficient unified benchmarks and irreproducible private data
• LLM-as-judge evaluation bias; offline/online metric gaps
• Latency, cost, hallucination, and safety constraints in deployed systems
• Cross-lingual and multimodal generalization

Practical Takeaways

1. Architecture: Cascade retrieval + rerank + generation remains mainstream; agentic paradigms make retrieval strategy itself learnable. 2. Data: High-quality instruction data and click/session logs matter; synthetic data requires leakage safeguards. 3. Engineering: Evaluate index update frequency, embedding versioning, p99 latency budgets, and failure modes (empty retrieval, wrong tool calls, over-generation) beyond paper metrics.

Glossary

| Term | Meaning | |------|---------| | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Gen-IR | Generative Information Retrieval | | Agentic Search | Search modeled as sequential decision-making with tool calls |