Learning To Rank Diversely At Airbnb (CIKM 2023)

Overview

This entry indexes an academic paper presented at CIKM 2023: *Learning To Rank Diversely At Airbnb*.

• Paper link: https://dl.acm.org/doi/abs/10.1145/3583780.3614692
• DOI: 10.1145/3583780.3614692
• Type: Academic paper
• Category: Verticals (search / ranking / recommendation)

> Note: The forum post is primarily a metadata index with general context. It does not reproduce the paper's quantitative results; consult the original ACM paper for details.

Key points

• The paper addresses learning-to-rank (LTR) problems at Airbnb, with a focus on diversity-aware ranking in large-scale search and personalization systems.
• Context: ranking systems face long-standing challenges in efficiency, scalability, and user intent understanding; traditional pipelines that separate retrieval, ranking, and generation are increasingly strained in the LLM era.
• Relevant trend lines discussed by the post: the evolution from BM25 and dense two-tower retrieval to cross-encoder reranking, late interaction, generative retrieval, and LLM-based agentic search, each balancing efficiency, effectiveness, and maintainability.
• Engineering considerations highlighted for production ranking systems:
• Latency: p99 budgets, cascaded retrieval with early stopping, caching, async reranking
• Data: PII handling, embedding version management, avoiding leakage in synthetic data
• Evaluation: gaps between offline metrics (nDCG@10, MRR, Recall@k) and online satisfaction; cross-validating LLM-as-judge with human evaluation
• Safety & cost: retrieval poisoning/bias mitigation, per-query token and GPU cost control
• The entry cross-references related industrial work on query–product relevance labeling with LLMs, ads conversion prediction, e-commerce behavior modeling, and cross-lingual LLM evaluation.

Related entries

• An Interpretable Ensemble of Graph and Language Models
• Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery
• Automated Query-Product Relevance Labeling using Large Language Models
• Behavior Modeling Space Reconstruction for E-Commerce Search
• Better to Ask in English: Cross-Lingual Evaluation of Large Language Models

Takeaways

1. Architecture: cascade retrieval + rerank + generation remains mainstream; agentic paradigms increasingly treat retrieval policy itself as learnable. 2. Data: high-quality instruction data and click/session logs remain critical alongside model innovations. 3. Evaluation: offline ranking metrics and online user satisfaction are diverging; validate with human audits. 4. Deployment: latency, cost, explainability, and safety are hard constraints that cannot be ignored in favor of benchmark gains.

References

• Original paper: *Learning To Rank Diversely At Airbnb*, CIKM 2023 — ACM DL