Fine-Tuning LLaMA for Multi-Stage Text Retrieval (RepLLaMA / RankLLaMA)

Overview

This forum post summarizes the arXiv paper Fine-Tuning LLaMA for Multi-Stage Text Retrieval (arXiv:2310.08319, October 2023) by Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin.

The paper shows that a pretrained LLaMA-2 model, when simply fine-tuned on supervised data, can power both stages of a modern multi-stage retrieval pipeline:

• RepLLaMA — a dense first-stage retriever that uses the LLaMA representation for passage embeddings;
• RankLLaMA — a second-stage pointwise reranker built on the same base model.

Notably, this is achieved without instruction tuning or distillation, and the resulting models outperform strong baselines on BEIR and TREC Deep Learning benchmarks.

Context and Motivation

Neural information retrieval has evolved from BM25 through BERT cross-encoders and dual-tower dense retrievers to late-interaction, generative retrieval, and LLM-based agents. Each generation balances the effectiveness–efficiency–maintainability triangle:

• Dense retrieval enables millisecond-level recall via approximate nearest-neighbor search but is sensitive to domain shift and long-tail queries;
• Cross-encoders are accurate but cannot precompute document representations;
• LLM-based approaches bring strong semantic priors, raising the question of whether pretrained decoder-only LLMs can directly replace BERT-style architectures in retrieval pipelines.

Method Sketch

The typical pipeline follows: problem formalization → model design → training → inference.

1. Input and representation: encode queries and passages using the LLaMA backbone (dense embeddings for retrieval; query-document concatenation for reranking); 2. Learning strategy: supervised fine-tuning on MS MARCO training pairs; 3. Inference pipeline: RepLLaMA performs first-stage recall over the full corpus, and RankLLaMA reranks the top candidates.

Evaluation

Typical evaluation in this line of work uses:

• Datasets: MS MARCO (training and in-domain evaluation), BEIR, TREC Deep Learning;
• Metrics: nDCG@10, MRR, Recall@k;
• Baselines: BM25, dense retrievers, cross-encoder rerankers.

Refer to the original PDF for exact numerical results.

Takeaways for Search / Rec / Personalization

1. Architecture: cascade retrieval + reranking + generation remains mainstream; pretrained LLMs can serve as strong backbones for both retriever and reranker; 2. Data: high-quality supervised data (e.g., MS MARCO) matters more than elaborate prompting; 3. Deployment: latency, cost, and safety are hard constraints — consider early stopping, query caching, and distillation for production; 4. Open problems: evaluation reliability, cross-lingual generalization, and index update costs remain open.

Related Entries

• CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval
• CoEvo: Coevolution of LLM and Retrieval Model for Domain-Specific Information Retrieval
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
• ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance
• How Does Generative Retrieval Scale to Millions of Passages?
• How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |

Reference

• Original paper: Fine-Tuning LLaMA for Multi-Stage Text Retrieval — https://arxiv.org/abs/2310.08319