LLMs for User Interest Exploration in Large-scale Recommendation Systems (Google, KDD 2024 GenAIRecP Workshop)
This post indexes a Google research paper presented at the Generative AI and Recommender Systems (GenAIRecP) Workshop at KDD 2024:
- Title: LLMs for User Interest Exploration in Large-scale Recommendation Systems
- Affiliation: Google
- Paper PDF: https://genai-personalization.github.io/assets/papers/GenAIRecP2024/Wang.pdf
- Section: Discovery
- Retrieval: BM25 → BERT cross-encoders → dense two-tower retrieval → late interaction → generative retrieval and LLM agents.
- Recommendation: matrix factorization → deep CTR models → sequential Transformers → LLM instruction-following and generative recommendation (Gen-Rec).
- RAG / Agentic Search: external knowledge access becomes iterative, verifiable, and plannable; evaluation shifts from static nDCG toward task success rate, citation accuracy, and multi-hop reasoning chain completeness.
- Original paper PDF: <https://genai-personalization.github.io/assets/papers/GenAIRecP2024/Wang.pdf> (GenAIRecP Workshop @ KDD 2024, Google)
Context and Motivation
Large-scale search and recommendation systems have long faced challenges around efficiency, scalability, and understanding user intent. Traditional pipeline approaches separate retrieval, ranking, and generation, which makes it hard to meet modern expectations for natural-language interaction, multi-hop reasoning, and fresh knowledge. This work explores how LLMs can be used to explore and expand user interests in an industrial-scale recommender, complementing conventional retrieval with semantic priors and cold-start capability.
The core tension highlighted in the post: LLMs bring strong semantic understanding, but online inference cost, latency budgets, and hallucination risks require careful system design when item catalogs are massive and user behavior is sparse.
Where It Fits Technically
The work sits at the intersection of recommendation and large-scale search. In the classic search stack — recall (coverage) → ranking (discrimination) → generation (presentation) — the LLM era adds two new variables: the inference budget and the action space (whether to retrieve, how many times, and which tools to call).
Related evolution across the field:
Engineering Deployment Checklist
| Area | Question | Suggestion | |--------|----------|------------| | Data | Does training/indexing contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascades with early stopping, caching hot queries, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source allow-lists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Route to smaller models, distillation, hybrid sparse+dense retrieval |
Practical Advice
1. Researchers: Reproduce the core comparisons; check whether statistical significance and compute costs are reported. 2. Engineers: Extract pluggable modules (encoders, re-rankers, planners) and evaluate integration cost with your existing stack. 3. Product managers: Focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.
Treat paper conclusions as directional evidence: distribution drift, seasonality, new-item cold start, and multilingual user mixtures can change optimal strategies. Validate with small-scale shadow experiments and A/B tests before full rollout.
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calls | | Gen-IR | Generative Information Retrieval |