How Does Generative Retrieval Scale to Millions of Passages? (Google Research, arXiv 2305.11841)
This forum entry covers the Google Research paper "How Does Generative Retrieval Scale to Millions of Passages?" (May 2023).
- Paper: https://arxiv.org/abs/2305.11841
- Authors: Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, et al. (8 authors total)
- Category: Information Retrieval / Generative Retrieval
- Problem: Generative retrieval—where a sequence-to-sequence model directly generates document identifiers instead of scoring a fixed index—had only been shown on small corpora. It was unclear whether it could scale to realistic web-scale settings with millions of passages.
- Setup: The authors evaluate on the MS MARCO training set, comprising 8.8M passages, using a T5-based seq2seq architecture that generates document identifiers autoregressively.
- Scaling challenges identified:
- Naive docid identifiers (e.g., random tokens) fail to generalize at scale because they carry no semantic signal.
- Beam search can produce invalid or duplicate identifiers, degrading recall.
- Model capacity and identifier design jointly determine scalability.
- Proposed solutions:
- Semantic identifiers derived from semantic clustering of passages, so similar documents share identifier prefixes and the model can generalize to unseen queries.
- Model scaling: increasing model size substantially improves generative retrieval quality at large corpus sizes.
- Constrained beam search to restrict decoding to valid identifier prefixes, eliminating invalid predictions.
- Findings: With semantic IDs, adequate model scale, and constrained decoding, generative retrieval becomes competitive with strong dense retrieval baselines (e.g., dual-encoder approaches) on MS MARCO-scale data—an important proof of viability for generative IR beyond toy benchmarks.
- Generative retrieval can serve as an alternative or complement to first-stage dense retrieval, potentially simplifying the retrieve-then-rerank cascade.
- Key engineering constraints remain: index update costs, latency/token budgets, and evaluation alignment between offline metrics (MRR, Recall@k) and online user satisfaction.
- Open questions include scaling beyond ~10M passages, multi-lingual coverage, and combining generative retrieval with rerankers and LLM-based generation (RAG).
- CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval
- CoEvo: Coevolution of LLM and Retrieval Model for Domain-specific IR
- DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval (arXiv 2310.08319)
Key points
Context in the IR landscape
The paper sits in the evolution of neural retrieval: from BM25 → BERT cross-encoders → dense bi-encoders with ANN search → late interaction → generative retrieval. Dense retrieval offers millisecond recall via approximate nearest-neighbor search but is sensitive to domain shift; cross-encoders are accurate but cannot precompute document representations; generative approaches remove cascade errors but face index-update and identifier-design challenges. This work addresses the identifier and scaling bottleneck of the generative paradigm.