Transforming Location Retrieval at Airbnb: From Heuristics to Reinforcement Learning (CIKM 2024)

This post summarizes the CIKM 2024 industry paper "Transforming Location Retrieval at Airbnb: A Journey from Heuristics to Reinforcement Learning."

• Original source: ACM Digital Library — DOI 10.1145/3627673.3680089
• Venue: CIKM 2024 (ACM Conference on Information and Knowledge Management), Industry Track

Background and Motivation

In large-scale search, recommendation, and personalization systems, information retrieval has long faced challenges around efficiency, scalability, and understanding user intent. Traditional pipeline architectures split retrieval, ranking, and generation into separate stages, which can struggle to meet the demands of the LLM era: natural language interaction, multi-hop reasoning, and up-to-date knowledge.

Airbnb's paper addresses this in a concrete industrial setting: location retrieval — deciding which destinations (cities, regions, neighborhoods) a guest's travel search should match, even when queries contain colloquial names, multiple destinations, or spelling errors. Historically this stage relied on hand-tuned heuristics; the paper describes the journey toward a reinforcement learning formulation.

Core Contributions

• A unified view of the location retrieval problem, consolidating previously scattered heuristics into a comparable framework.
• A clear decomposition of system components (representation learning, retriever, reranker, feedback mechanisms) to support engineering iteration.
• An account of how reinforcement learning reframes retrieval strategy itself as a learnable sequential decision problem in production.
• Discussion of interfaces with emerging paradigms such as LLM tool calling and agentic search, including paths from research prototypes to industrial systems.
• Explicit open problems: evaluation reliability, latency and cost, hallucination and safety, and cross-lingual/multimodal expansion.

Typical Method Architecture

The general technical pipeline follows four stages: problem formalization → model/system design → training or construction → inference pipeline.

1. Input and representation: encode queries, documents, and user context as dense/sparse representations or structured prompts. 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, chained or combined per task. 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrapped data synthesis. 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.

Evaluation Considerations

For this class of systems, evaluation typically involves:

• Datasets: benchmarks such as MS MARCO, BEIR, Natural Questions, plus proprietary industry corpora.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.

Precise quantitative results should be read from the original PDF tables.

Key Takeaways for Search / Rec / Personalization

1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms make the *retrieval policy itself* a learnable object. 2. Data: high-quality instruction data and click/session logs are critical; synthetic data must guard against leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product constraints: latency, cost, explainability, and safety policies are hard requirements for production deployment — not just academic benchmarks.

Limitations and Future Work

Likely limitations include experiment scale constrained by compute budgets, benchmarks that mismatch real user distributions, English-centric data leaving cross-lingual generalization untested, and safety risks of agentic systems operating on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommenders.

Related Entries

• An interpretable ensemble of graph and language models
• Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery
• Automated Query-Product Relevance Labeling using Large Language Models
• Behavior Modeling Space Reconstruction for E-Commerce Search (arXiv 2501.18216)
• Behavior-driven query similarity prediction based on pre-trained language models
• Better to Ask in English: Cross-Lingual Evaluation of Large Language Models (DOI 10.1145/3589334.3645643)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool use | | Gen-IR | Generative Information Retrieval |