REFRAG: Meta and NUS Collaborate on an Efficient Decoding Framework for RAG

Forum topic · ✨步子哥 · 2025-12-04

Summary

REFRAG is an efficient decoding framework for Retrieval-Augmented Generation (RAG) developed jointly by Meta Superintelligence Labs, the National University of Singapore, and Rice University. It addresses the quadratic attention cost, KV cache memory growth, and information sparsity that occur when retrieved documents are fed to large language models. REFRAG exploits the block-diagonal sparsity pattern across retrieved passages using a three-stage design: Compress (chunks are embedded into single vectors by a lightweight encoder such as RoBERTa), Sense (a projection MLP aligns embeddings with the main LLM's token space), and Expand (reinforcement learning selects which chunks to decompress to preserve precise details like numbers and dates). Benchmarks report up to 30.85x first-token latency speedup (k=32), 16x longer contexts, 6.78x throughput improvement, and roughly k-fold KV cache reduction, with accuracy on 16 RAG tasks matching or exceeding full-context LLaMA and 9.3% lower perplexity than the CEPE baseline on Book and Arxiv datasets. Paper: https://arxiv.org/abs/2509.01092

Overview

REFRAG is an efficient decoding framework for Retrieval-Augmented Generation (RAG), developed in collaboration between Meta Superintelligence Labs, the National University of Singapore (NUS), and Rice University. Lead authors include NUS PhD student Lin Xiaoqiang.

The Long-Context Challenge in RAG

Time cost: attention complexity grows quadratically with sequence length, causing very high time-to-first-token (TTFT).
Space cost: the KV cache grows linearly with sequence length, limiting batch size and throughput.
Information sparsity: among dozens of retrieved documents, only a few snippets are truly relevant to the query; the remaining tokens contribute little yet still participate in all attention computations.

Block-Diagonal Attention Sparsity

Retrieved passages tend to have low semantic correlation with each other, producing a block-diagonal sparsity pattern in attention when the model generates tokens.
Tokens within a passage are strongly related; tokens across passages are weakly related.
Feeding all raw tokens into the LLM is therefore unnecessary and inefficient.

Three-Stage Design: Compress, Sense, Expand

1. Compress — a lightweight encoder (e.g., RoBERTa) compresses text chunks into single embedding vectors, greatly reducing sequence length. 2. Sense — a projection layer (MLP) maps encoder embeddings into the main LLM's token embedding space, aligning the two models. 3. Expand — a reinforcement-learning policy intelligently selects which chunks to decompress, preserving key details (exact numbers, dates) that compression would otherwise lose.

Measured Results

| Metric | Value | |---|---| | First-token speedup (k=32) | 30.85x | | Context length extension | 16x | | Throughput improvement | 6.78x | | KV cache memory reduction | ~k-fold |

Accuracy on 16 RAG tasks matches or exceeds the full-context LLaMA model.
Perplexity (PPL) on Book and Arxiv datasets is on average 9.3% lower than the CEPE baseline.

Applications

Enterprise knowledge-base Q&A — large-scale document retrieval with fast responses.
Multi-turn dialogue — no need to truncate history while preserving context.
Long-document summarization — books, reports, and other very long texts.
Agent applications — complex reasoning and tool use.

Reference

Paper: https://arxiv.org/abs/2509.01092