INTRA: Attention Itself Is the Retriever — NVIDIA Rethinks the RAG Paradigm

Forum topic · 小凯 · 2026-06-10

Summary

NVIDIA and Technion researchers propose INTRA (Retrieval from Within), a framework showing that cross-attention in encoder-decoder Transformers is itself a retrieval mechanism, eliminating the need for external retrievers like BM25 or dense embedding models. By appending 64 trainable retrieval tokens and training only ~164K parameters on a frozen T5Gemma2 backbone, INTRA performs coarse encoder-side MaxSim retrieval followed by fine-grained decoder-side scoring that aggregates query states across layers. A Reverse-QWK trick stores a single normalized encoder representation so all decoder layers can share one pre-encoded evidence pool. On HotPotQA, INTRA reaches 59.9% R@5, beating BGE and Qwen3-Embedding-4B + Jina reranker baselines by over 5 points, and leads on 2WikiMultihopQA and MuSiQue as well, with the best end-to-end exact match on all multi-hop benchmarks. The paper argues the gap-closure comes from evidence assembly: decoder queries know what the generator needs. Pre-encoded chunks are reused across queries, matching standard RAG compute costs. Limitations include QA-only evaluation and a relatively weak backbone.

> Paper: *Retrieval from Within: An Intrinsic Capability of Attention-Based Models* > Authors: Elad Hoffer, Yochai Blau, Edan Kinderman, Ron Banner, Daniel Soudry, Boris Ginsburg > Institutions: NVIDIA + Technion > Paper: https://arxiv.org/abs/2605.05806

Key Points

Core claim: Attention is already a query-conditioned matching operation — cross-attention is mathematically equivalent in spirit to retrieval scoring (MaxSim). No external retriever is needed.
Method: A frozen encoder-decoder (T5Gemma2 4B) with 64 trainable retrieval tokens and only ~164K trainable parameters performs:

Encoder-side coarse retrieval (S₀)

Decoder-side fine retrieval (S_INTRA)

sᵢ = Σ_ℓ α_ℓ · MaxSim(q_ℓ, k_i)

Generation

Reverse-QWK engineering trick: Instead of per-layer key projections (which break index reuse), the key-side RMSNorm + projection is moved to the query side: q̃_ℓ = (q_ℓ W_K,ℓ^T) ⊙ γ_K,ℓ. A single normalized encoder pool serves all decoder layers and supports GQA.
Training: Soft cross-entropy on oracle evidence chunks; the generator backbone stays frozen.

Results

Shared pool: ~100M tokens, 758,500 chunks. Baselines include TF-IDF, BM25, BGE-large, Qwen3-Embedding-0.6B/4B, Jina reranker, hybrid RRF, ColBERT-style MaxSim.

Full-evidence recall R@5:

| Dataset | Best baseline | INTRA | Δ | |---|---|---|---| | HotPotQA | 54.8 (BGE) | 59.9 | +5.1 | | 2WikiMultihopQA | 35.4 | 40.7 | +5.3 | | MuSiQue | 10.1 | 12.8 | +2.7 | | Natural Questions | 31.9 | 29.1 | −2.8 |

End-to-end EM (average): INTRA 36.7 vs. BGE 34.4; first place on all multi-hop benchmarks (HotPotQA 51.3, 2Wiki 47.3, MuSiQue 8.5).

Ablations: full-corpus decoder scoring recovers evidence missing from S₀ (37.1% → 59.9% R@5 on HotPotQA); 64 retrieval tokens and chunk length L_p=7 are the sweet spots.

Why It Wins on Multi-Hop QA

Multi-hop questions require evidence assembly across chunks. Decoder queries know what the generator needs, so retrieval signals come from the generation process itself. On single-hop NQ, where one passage suffices, traditional retrievers are already good and INTRA shows no advantage.

The paper's GapClosure metric shows that sharing the same decoder for retrieval and generation closes the largest gap between random-chunk and complete-evidence performance — evidence that unified representations matter.

Efficiency

Compute matches standard RAG (O(k·L_c·L_q) prefill), but INTRA reuses pre-encoded chunk representations — no re-encoding after retrieval. A 1B-token corpus at 8-bit quantization costs ~2.5TB to store.

Implications

RAG's split retriever/generator design (mismatched representation spaces, double encoding, retrieval blind to generation needs) may be suboptimal.
Pretrained models already contain retrievable capabilities; only ~164K parameters are needed to unlock them.
Encoder-decoder architectures have unique advantages for retrieval-augmented generation.

Limitations

1. Evaluated only on QA; no code generation or long-document summarization. 2. Backbone (T5Gemma2 4B) is weaker than frontier decoder-only LLMs. 3. Only Wikipedia-style corpora tested. 4. Retrieval training requires labeled oracle evidence.

Key Quote

> "Attention-based models already possess a retrievable, elicitable retrieval mechanism — rather than requiring one to be added as an external module."

References

Hoffer, E. et al. (2026). Retrieval from Within: An Intrinsic Capability of Attention-Based Models. arXiv:2605.05806.
Thakur, N. et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.