Paper Overview
Field: ML Authors: Jonathan Kozaczuk, Zheng Dong, Noyan Evirgen, Keivan Majidi, Eddie K. H. Ng, Chris Porter, Qi Zhao, Jia Geng, Changmao Li, Brandon R. Feng Published: 2026-11-05 arXiv: 2611.08487
Abstract
User embeddings are central components of modern recommendation, personalization, and behavioral modeling systems. Though these embeddings can compactly encode rich behavioral information, they are inherently dense and opaque, limiting interpretability. To address this gap, we argue that user embeddings can be understood as sparse superpositions of interpretable behavioral modes. This structure is directly predicted by generative models of user behavior for linear embeddings and extends approximately to deep-learning representations by the superposition hypothesis. Superimposed modes can naturally be recovered by sparse dictionary learning methods, such as sparse autoencoders (SAEs), and recovered modes can be automatically named and queried in natural language. We validate mode recovery against known ground truth on synthetic data, where SAEs outperform classical sparse baselines, including on Transformer-encoded embeddings. We then apply the method to classic and Transformer-based user embeddings from two real-world datasets. We also demonstrate how the corresponding natural-language labels enable zero-shot audience retrieval and profiling on embeddings never trained on text. The framework is post-hoc and model-agnostic, requiring no modification to upstream embedding systems.
Key Contributions
- Interpretability framing: User embeddings are treated as sparse superpositions of interpretable behavioral modes.
- Theoretical grounding: The structure is predicted by generative models of user behavior (linear case) and extends to deep representations via the superposition hypothesis.
- Recovery method: Sparse autoencoders (SAEs) recover superimposed modes, outperforming classical sparse baselines in synthetic validation with known ground truth.
- Natural-language modes: Recovered modes are automatically named and queryable, enabling zero-shot audience retrieval and profiling on embeddings never trained on text.
- Practical applicability: Demonstrated on classic and Transformer-based user embeddings from two real-world datasets.
- Drop-in design: The framework is post-hoc and model-agnostic, requiring no changes to upstream embedding systems.
*Auto-collected on 2026-11-07*