REFRAG: Meta & NUS Efficient Decoding Framework for RAG Delivers 30.85x TTFT Speedup

Forum topic · ✨步子哥 · 2025-12-04

Summary

REFRAG is an efficient decoding framework for retrieval-augmented generation (RAG) developed through collaboration between Meta Superintelligence Labs, the National University of Singapore, and Rice University. It addresses the quadratic attention cost, KV cache memory growth, and information sparsity that plague long-context RAG workloads. Exploiting the block-diagonal sparsity pattern of retrieved passages, REFRAG uses a three-stage design: Compress (a lightweight encoder such as RoBERTa compresses text chunks into single embedding vectors), Sense (an MLP projection aligns these vectors into the main LLM's token space), and Expand (a reinforcement learning policy selectively decompresses key chunks to preserve details like exact numbers and dates). Reported results include up to 30.85x faster time-to-first-token at k=32, 16x longer context, 6.78x throughput improvement, and roughly k-fold KV cache reduction, while matching or exceeding full-context LLaMA accuracy across 16 RAG tasks and lowering perplexity by an average of 9.3% versus CEPE on Book and Arxiv datasets. Paper: https://arxiv.org/abs/2509.01092

REFRAG: An Efficient Decoding Framework for RAG

REFRAG is a collaboration between Meta Superintelligence Labs, the National University of Singapore, and Rice University (lead author: NUS PhD student Lin Xiaoqiang and colleagues).

Paper: https://arxiv.org/abs/2509.01092

The Long-Context Challenge in RAG

Time cost: attention complexity grows quadratically with sequence length, causing very high time-to-first-token (TTFT).
Space cost: the KV cache grows linearly with sequence length, limiting batch size and throughput.
Information sparsity: among dozens of retrieved documents, only a few fragments are truly relevant to the query; the rest contribute almost nothing to generation yet still participate in all attention computation.

Block-Diagonal Attention Sparsity

Retrieved passages in RAG tend to be semantically weakly related to each other, producing a block-diagonal sparsity pattern: tokens within one passage are strongly correlated, while tokens across passages are only weakly related. Feeding all raw tokens to the LLM is therefore unnecessary and inefficient.

Core Design: Compress–Sense–Expand

1. Compress: a lightweight encoder (e.g., RoBERTa) compresses text chunks into single embedding vectors, drastically reducing sequence length. 2. Sense: an MLP projection layer maps encoder embeddings into the main LLM's token embedding space, aligning the two models. 3. Expand: a reinforcement learning policy intelligently selects which information chunks to decompress, ensuring key details (exact numbers, dates) are not lost to compression.

Results

| Metric | Result | |---|---| | TTFT speedup (k=32) | 30.85x | | Context length extension | 16x | | Throughput improvement | 6.78x | | KV cache reduction | ~k-fold |

Accuracy on 16 RAG tasks is comparable to or higher than a full-context LLaMA model.
Average perplexity (PPL) reduction of 9.3% versus the CEPE baseline on Book and Arxiv datasets.

Applications

Enterprise knowledge-base Q&A with large-scale retrieval
Multi-turn dialogue without truncating history
Long-document summarization (books, reports)
Agent applications with complex reasoning and tool use