Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (Meta, arXiv 2402.17152)

Forum topic · 小凯 · 2026-07-05

Summary

This February 2024 arXiv paper (2402.17152) from Meta introduces the Generative Recommenders framework built on the Hierarchical Transduction Unit (HSTU), a new attention-based architecture designed to replace the longstanding dominant paradigm of feature-crossing deep learning models in recommender systems. The authors argue that recommendation should be treated as a generative modeling problem over sequences of user actions rather than static features. HSTU reformulates the Transformer with pointwise-aggregated attention and uses discretized tokens for both items and user actions, enabling scaling to trillions of parameters. In experiments spanning a 10-billion-item testbed and a large-scale production dataset, the model outperforms strong baselines by 24.5% to 65.8% on relevant metrics (such as NDCG@10 for recommendation tasks). When deployed as the core ranking model in a real online A/B test at Meta, serving 200 million users across 20+ surfaces, the generative recommender delivered up to 12.4% improvements in online engagement metrics. The paper also shows scaling-law behavior: performance follows power-law scaling with model size and compute, analogous to LLM pretraining trends. Training efficiency is improved through a new relative positional encoding and attention formulation that reduces latency and supports very long user histories. The work represents one of the first demonstrations of trillion-parameter models successfully powering industrial-scale recommendation serving.

Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

Paper: https://arxiv.org/abs/2402.17152 (arXiv, February 2024) Authors: Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Lei Gao, Zhaojie Gong, Saikat Choraria, Lei Ji, Simon Wang, Daisy Zhu, Anima Anandkumar (Meta AI & collaborators)

Key points

New architecture — HSTU (Hierarchical Transduction Unit): A Transformer-based sequential transducer that replaces feature-crossing networks (FCNPs) that have dominated recommendation ranking since 2013. HSTU modifies the self-attention formulation with a pointwise-aggregated attention mechanism and a simplified relative positional encoding, improving both quality and efficiency.
Generative framing of recommendation: The system treats recommendation as a generative sequence modeling task, encoding user actions and content as discrete tokens, and predicting the next action in a user's timeline — directly analogous to autoregressive language modeling.
Trillion-parameter scale: The authors demonstrate scaling laws in recommendation: model quality follows a power law with respect to model size and compute. They train and serve models with up to a trillion parameters.
Offline results: On a testbed covering 10 billion items plus a large public dataset, HSTU-based generative recommenders outperform strong baselines by 24.5%–65.8% on NDCG@10-style ranking metrics.
Online deployment: The model was deployed in a real online A/B test at Meta, serving over 200 million users across more than 20 product surfaces, delivering up to 12.4% improvements in online engagement — a rare case of a generative architecture validated at industrial scale.
Efficiency contributions: The modified attention and positional encoding reduce latency and memory cost, enabling very long user histories (up to tens of thousands of tokens) to be processed for serving.

Method outline

1. Tokenization: Items, user actions, and contextual signals are converted into discrete tokens over a unified vocabulary. 2. Sequence modeling: HSTU layers process chronological user histories; attention aggregates pointwise signals rather than pairwise-only interactions, improving compute efficiency. 3. Training: Autoregressive next-action prediction objectives, following scaling-law guidance similar to LLM pretraining. 4. Inference: A ranking stage samples and scores candidate next actions from the generative model, integrated into an existing retrieval-and-ranking serving stack.

Why it matters

This is one of the first published works to show that (a) scaling laws transfer to industrial recommender systems, and (b) a generative Transformer-based ranker can replace legacy feature-crossing production models and win in live A/B tests at very large scale. It bridges the gap between LLM-era sequence modeling and the recommendation domain, where user behavior is sparse, catalogs are enormous, and latency budgets are strict.

Related entries

360Brew: A Decoder-only Foundation Model for Personalized Ranking
Bridging Language and Items for Retrieval and Recommendation (BLAIR), Mar 2024
EAGER-LLM: Enhancing LLMs as Recommenders through Exogenous Features

Cite as: Zhai et al., *Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations*, arXiv:2402.17152, 2024.