Text Is All You Need: Learning Language Representations for Sequential Recommendation (arXiv:2305.13731)
Source: https://arxiv.org/abs/2305.13731 · arXiv, May 2023 Authors: Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, et al. (7 authors in total)
Overview
This paper proposes Recformer, a sequential recommendation framework that uses text as the sole representation of items. Rather than learning large embedding tables for item IDs, the model treats each item as a piece of text (title, brand, category, and attribute keywords) and a user's interaction history as a sequence of such texts — effectively casting sequential recommendation as a language understanding problem.
Key Ideas
- Items as text: Every item is described by its textual metadata, so no ID embeddings are needed. This makes the approach naturally suited to cold-start scenarios, where new items have no interaction history but do have descriptions.
- Language model backbone: Recformer builds on a Longformer-style transformer encoder that can handle long concatenated item-text sequences representing a user's behavior history.
- Text-contrastive pretraining: The model is pretrained with a contrastive learning objective designed to pull semantically and behaviorally similar items close together in the representation space, aligning the language encoder with recommendation needs.
- Next-item prediction as text selection: At fine-tuning and inference time, the model scores candidate items by their textual fit with the user's history "sentence," selecting the next item accordingly.
- Unified recipe: Recommendation inherits the generalization of pretrained language models, reducing the need for task-specific item ID architectures.
- Cold-start friendly: New and long-tail items can be recommended immediately from metadata alone.
- Industry relevance: Co-authored by Amazon researchers, the work reflects production interest in language-model-driven personalization.
- 360Brew: A Decoder-only Foundation Model for Personalized Ranking
- Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers
- Bridging Language and Items for Retrieval and Recommendation (Mar 2024)
- Data-efficient Fine-tuning for LLM-based Recommendation, SIGIR 2024
- DiffKG: Knowledge Graph Diffusion Model for Recommendation, WSDM 2024
- Li, J., Wang, M., Li, J., Fu, J., Shen, X., Shang, J., et al. *Text Is All You Need: Learning Language Representations for Sequential Recommendation.* arXiv:2305.13731, May 2023. https://arxiv.org/abs/2305.13731
Reported Results
The authors evaluate on public sequential recommendation benchmarks and report that Recformer outperforms competitive ID-based and text-based baselines, while transferring well to items unseen during training thanks to its text-based representation. Exact numbers should be verified against the paper's tables.