Fine-Tuning LLaMA for Multi-Stage Text Retrieval
- Paper: Fine-Tuning LLaMA for Multi-Stage Text Retrieval
- Authors: Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
- Venue: arXiv preprint, October 2023
- Area: Information Retrieval / Neural Ranking
- RepLLaMA — a dense (bi-encoder) retriever for first-stage candidate generation, trained with contrastive learning on MS MARCO.
- RankLLaMA — a pointwise reranker (used as a cross-encoder) that re-ranks the retrieved candidates.
- Prior LLM-based retrieval research mostly used LLMs to generate synthetic training data or relevance labels; this paper instead shows the LLM itself can be an effective retrieval backbone.
- The same pretrained model family is used for both retrieval stages, yielding a unified pipeline without the usual retriever/reranker architecture mismatch.
- On MS MARCO and BEIR, the fine-tuned LLaMA retriever and reranker outperform strong baselines, including widely used dense retrievers (e.g., based on BERT-style encoders) and their rerankers.
- The pointwise reranking formulation with LLaMA is simpler than listwise or setwise approaches while remaining highly effective.
- Fine-tuning is done efficiently with LoRA-style parameter updates, making the approach practical on academic hardware budgets.
- Evaluations are primarily on public academic benchmarks (MS MARCO, BEIR); production-scale latency and cost analysis is not the focus.
- Exact metrics should be verified against the original PDF tables.
- CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval
- CoEvo: Coevolution of LLM and Retrieval Model for Domain-Specific Information Retrieval
- DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
- ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance
- How Does Generative Retrieval Scale to Millions of Passages? (Google Research)
- Original paper: <https://arxiv.org/abs/2310.08319>
Summary
This work investigates whether LLaMA can replace conventional BERT-style transformer models across the entire multi-stage retrieval pipeline. The authors fine-tune LLaMA in two roles:
Key points
Implications
1. Unified pipelines: Using one LLM for both stages reduces engineering complexity compared with mixing bi-encoders and cross-encoders of different architectures. 2. Open-weight advantage: Results strengthen the case for open-weight moderate-size LLMs in production search stacks. 3. Baseline shift: Later work on LLM-based retrieval and reranking (e.g., instruction-following retrievers) frequently cites this paper as a foundational result.