LLMs for User Interest Exploration in Large-scale Recommendation Systems
Work by Google, presented at the Generative AI and Recommender Systems (GenAIRecP) Workshop at KDD 2024.
Paper PDF: https://genai-personalization.github.io/assets/papers/GenAIRecP2024/Wang.pdf
About this entry
This forum post catalogs a Google research paper on applying LLMs to user interest exploration in large-scale recommendation systems. Much of the post is a standard catalog template, so this page preserves the verifiable facts and the surrounding discussion.
Context and motivation
Large-scale search, recommendation, and personalization systems have long faced challenges in efficiency, scalability, and understanding user intent. Traditional pipeline designs tend to separate retrieval, ranking, and generation, which makes it hard to meet users' combined expectations for natural-language interaction, multi-hop reasoning, and up-to-date knowledge in the LLM era.
The paper addresses the core scenario of discovering and expanding user interests at industrial scale using large language models.
Where this work sits in the field
- Information retrieval evolution: BM25 → BERT cross-encoders → bi-encoder dense retrieval → late interaction → generative retrieval and LLM agents. Each generation trades off efficiency, effectiveness, and maintainability.
- Recommendation evolution: matrix factorization → deep CTR models → sequential Transformers → LLM instruction-following and generative recommendation (Gen-Rec). Key tensions: sparse user behavior, huge item catalogs, and multi-objective business goals.
- LLM role: LLMs offer semantic priors and cold-start capability, but online inference cost and hallucination risk demand careful system design.
- Original paper: *LLMs for User Interest Exploration in Large-scale Recommendation Systems*, GenAIRecP Workshop @ KDD 2024 (Google). PDF
Engineering checklist (from the post)
| Check | Question | Recommendation | |-------|----------|----------------| | Data | Does training/indexing contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascades + early stopping, cache popular queries, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source allow-lists, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Route to smaller models, distillation, hybrid sparse+dense retrieval |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |
Discussion
In industrial search and recommendation systems, model improvements must iterate jointly with index architecture, feature pipelines, and online serving. Methods validated on controlled benchmarks typically need recalibration in production: query distribution drift, seasonality, new-item cold start, and multilingual user mixes can significantly change the optimal strategy. Readers are advised to treat the paper's conclusions as directional evidence and validate via small-scale shadow experiments and A/B tests before full rollout.
Action suggestions for readers
1. Researchers: Reproduce the core comparison experiments; check whether statistical significance and compute cost are reported. 2. Engineers: Extract pluggable modules (encoders, re-rankers, planners) and assess integration cost with the existing stack. 3. Product managers: Focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.