Learning To Rank Diversely at Airbnb (CIKM 2023)
Paper: Learning To Rank Diversely At Airbnb, CIKM 2023 Source: https://dl.acm.org/doi/abs/10.1145/3583780.3614692 Type: Academic paper (CIKM 2023, Verticals track)
Key points
- The paper tackles diversified learning to rank (LTR) at Airbnb, targeting the tension between ranking purely by relevance and producing diverse result lists in a marketplace search setting.
- It belongs to the broader ranking research area, where large-scale search/recommendation systems face challenges in efficiency, scalability, and user-intent understanding.
- In industrial systems, overly homogeneous rankings can reduce long-tail item exposure and user satisfaction, motivating diversity-aware objectives alongside relevance.
- From BM25 to BERT-style cross-encoders, dense bi-encoder retrieval, late interaction, and now generative retrieval / LLM agents.
- Classic search stacks act as a funnel: recall for coverage, fine ranking for discrimination, generation for presentation. LLM-era systems add new variables: inference budget and action space (whether to retrieve, how often, which tools to call).
- On the recommendation side, the core tension is sparse user behavior, huge item catalogs, and multi-objective business goals; LLMs add semantic priors and cold-start help but raise online inference cost and hallucination risk.
- An interpretable ensemble of graph and language models for improving search
- Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery
- Automated Query-Product Relevance Labeling using Large Language Models
- Behavior Modeling Space Reconstruction for E-Commerce Search (Jan 2025)
- Better to Ask in English: Cross-Lingual Evaluation of Large Language Models
- Original paper: Learning To Rank Diversely At Airbnb, CIKM 2023 — ACM DL
Context within modern ranking research
The forum entry situates this work in the evolution of neural information retrieval:
Engineering deployment checklist (from the entry)
| Area | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stopping, hot-query caching, async re-ranking | | Quality | Does offline gain convert to online CTR/satisfaction? | Interleaving experiments, human audit samples, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |
Takeaways for search / rec / personalization
1. Architecture: cascade of retrieval + re-ranking + generation remains mainstream, but agentic paradigms make retrieval strategy itself learnable. 2. Data: high-quality instruction data and click/session logs both matter; synthetic data needs leakage and distribution-shift controls. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product: latency, cost, explainability, and safety are hard production constraints — academic benchmarks alone are not enough.