RAFT: Adapting Language Model to Domain Specific RAG (July 2024, OpenReview)

Overview

| Field | Content | |------|------| | Title | RAFT: Adapting Language Model to Domain Specific RAG | | Date | July 2024 | | Source | OpenReview forum | | Topic area | RAG |

What RAFT Does

RAFT (Retrieval Augmented Fine-Tuning) addresses the gap between general-purpose RAG and domain-specific deployments (e.g., enterprise knowledge bases, medical or legal corpora). Instead of relying on a frozen general LLM whose behavior with retrieved documents is unpredictable, RAFT fine-tunes the model so that it:

• Receives a question together with one oracle (relevant) document and several distractor documents that mimic imperfect retrieval results;
• Learns to ignore distractors and ground its answer in the relevant evidence;
• Is trained on reasoning chains derived from the oracle document, so the model explains *why* an answer follows from the retrieved context rather than memorizing answers.

This shifts the emphasis from storing domain knowledge in the weights toward robust context utilization and citation, which matters when retrieval quality varies.

Position in the RAG Landscape

The post situates RAFT within four main research threads in neural information retrieval and generation:

• Dense retrieval: high recall, low latency, suited to first-stage retrieval, but sensitive to domain shift and long-tail queries;
• Late interaction (e.g., ColBERT): higher precision at the cost of larger indexes;
• Generative IR: documents are "generated" directly as tokens/docids, simplifying cascades but complicating index updates;
• Agentic search: search modeled as sequential decisions, supporting multi-hop reasoning and self-reflection.

Timeline: BERT re-ranking and DPR (2019–2021) established neural retrieval; RAG and FreshLLMS (2022–2023) fused retrieval and generation; from 2024 onward, conversational/agentic search and generative recommendation expanded rapidly, with RL-trained search agents, Deep Research, and GraphRAG as emerging directions.

Evaluation Paradigm

Typical benchmarks and metrics discussed for this line of work include:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora;
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost;
• Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.

For domain-specific RAG systems specifically, evaluation is shifting from static ranking metrics toward task success rate, citation accuracy, and completeness of multi-hop reasoning chains. Exact numbers for RAFT should be taken from the original paper's tables.

Takeaways

1. Architecture: cascade retrieve → re-rank → generate remains mainstream, but agentic paradigms make "how many retrieval steps and when to retrieve" itself learnable; 2. Data: high-quality instruction data with distractor-aware training (as in RAFT) is as important as the retriever itself; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation; 4. Deployment: latency, cost, interpretability, and safety are hard constraints in production — academic benchmarks alone are insufficient.

Open Problems

• Lack of unified benchmarks; private domain data limits reproducibility;
• LLM evaluation bias; hallucination and safety in agentic retrieval;
• Cross-lingual and multimodal generalization (training data is largely English-centric);
• Cost-aware inference: efficient test-time compute allocation and deeper integration with knowledge graphs or structured databases.

Practical Checklist for Engineering Teams

| Item | Question | Recommendation | |------|----------|----------------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades with early stopping, query caching, async re-ranking | | Quality | Do offline gains transfer to online satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias via open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Tokens and GPU per query? | Small-model routing, distillation, hybrid sparse+dense retrieval |

Related Entries

• A Survey of Graph Retrieval-Augmented Generation (arXiv:2501.13958)
• A Survey on Retrieval-Augmented Text Generation for LLMs (arXiv:2404.10981)
• Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (arXiv:2501.09136)
• RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv:2502.11371)

Glossary

| Term | Meaning | |------|------| | RAG | Retrieval-Augmented Generation | | RAFT | Retrieval Augmented Fine-Tuning | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Treating search as sequential decision-making and tool use | | Gen-IR | Generative Information Retrieval |

> Reference: RAFT: Adapting Language Model to Domain Specific RAG, July 2024. See the OpenReview discussion for the full paper.