Mamba4Rec: Efficient Sequential Recommendation with Selective State Space Models
Overview
Mamba4Rec (arXiv:2403.03900, March 2024) explores the use of selective state space models — specifically the Mamba architecture — as an efficient backbone for sequential recommendation.
| Field | Content | |-------|---------| | Title | Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models | | Authors / Affiliations | Chengkai Liu, Jianghao Lin, Jianling Wang, Hanzhou Liu, James Caverlee | | Published | March 2024, arXiv | | Link | https://arxiv.org/abs/2403.03900 | | Category | Sequential Recommendation |
Motivation
Sequential recommendation systems must encode users' interaction histories to predict their next actions. Transformer-based models (e.g., BERT4Rec, SASRec) capture long-range dependencies well, but self-attention introduces:
- Quadratic complexity with respect to sequence length, limiting long-history modeling;
- High inference latency and memory usage, which is problematic for real-time industrial serving.
- Outperforms representative RNN- and Transformer-based baselines in accuracy metrics;
- Achieves better efficiency (training/inference speed and memory), especially as sequence length grows.
- State space models are a credible alternative to Transformers for recommendation sequence encoders.
- Linear-complexity sequence modeling is attractive for production recommenders with long user histories.
- Mamba4Rec is an early data point in the "SSMs for RecSys" research direction.
- BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations
- EAGER: Two-Stream Generative Recommender
- Efficient On-Device Session-Based Recommendation
- How to Index Item IDs for Recommendation Foundation Models (P5, SIGIR)
- LLMCDSR: Cross-Domain Sequential Recommendation with LLMs
- Multi-Behavior Sequential Transformer Recommender (SIGIR 2024)
- Original paper: Mamba4Rec on arXiv
Mamba's selective state space model (SSM) offers linear-time sequence modeling with input-dependent gating, making it a natural candidate for efficient user behavior sequence encoding.
Approach
Mamba4Rec replaces the Transformer encoder with a Mamba-based block stack for encoding item interaction sequences:
1. Item embedding of the interaction sequence; 2. Selective SSM layers with input-dependent parameters that dynamically decide what information to propagate or forget along the sequence; 3. Next-item prediction via scoring against the item vocabulary.
The key benefit is linear scaling with sequence length, enabling long-history modeling at lower computational and memory cost than self-attention.
Results
Per the paper, on standard sequential recommendation benchmarks, Mamba4Rec:
Exact numbers should be verified against the paper's tables.