Real-time Personalization Using Embeddings for Search Ranking at Airbnb (KDD 2018)
Overview
This post catalogs the KDD 2018 paper "Real-time Personalization using Embeddings for Search Ranking at Airbnb", available in the ACM Digital Library:
- Source: https://dl.acm.org/doi/abs/10.1145/3219819.3219885
- Type: Academic paper
- Category: Verticals / Search & Recommendation
- The work is positioned in the intersection of embeddings and large-scale search/recommendation systems.
- It relates to core questions about how to allocate responsibilities across retrieval, ranking, generation, and tool use — with inference budget and action space as key variables in modern systems.
- The classic search stack operates as a funnel: recall/coverage, fine-grained ranking, and presentation.
- Neural IR has evolved from BM25 → BERT cross-encoders → two-tower dense retrieval → late interaction → generative retrieval and LLM agents, each generation balancing efficiency–effectiveness–maintainability.
- Dense retrieval achieves millisecond-level recall via approximate nearest neighbor search but is sensitive to domain shift and long-tail queries.
- On the recommendation side, the core tension is sparse user behavior, huge item catalogs, and multi-objective business goals.
- IR — Information Retrieval
- RAG — Retrieval-Augmented Generation
- LTR — Learning to Rank
- nDCG — Normalized Discounted Cumulative Gain, a ranking quality metric
- Agentic Search — modeling search as sequential decision-making with tool calls
- Gen-IR — Generative Information Retrieval
- An interpretable ensemble of graph and language models for improving search
- Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery (arXiv:2502.10514)
- Automated Query-Product Relevance Labeling using Large Language Models (arXiv:2502.15990)
- Behavior Modeling Space Reconstruction for E-Commerce Search (arXiv:2501.18216)
- Behavior-driven query similarity prediction based on pre-trained language models
- Better to Ask in English: Cross-Lingual Evaluation of Large Language Models (DOI: 10.1145/3589334.3645643)
The paper addresses long-standing challenges in large-scale search, recommendation, and personalization systems where embeddings face efficiency, scalability, and user-intent-understanding constraints. It aims to advance the theory and practice at the intersection of embedding-based retrieval and real-time personalization.
Key points
Context in the field
Engineering checklist (from the post's appendix)
| Area | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascaded retrieval + early stopping, hot-query caching, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |