LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG
Paper: LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG Authors / Affiliation: Yijia Zheng, Marcel Worring Published: 2026-05-07 Source: https://arxiv.org/abs/2605.06285 Category: Agentic Search
TL;DR
LatentRAG moves both reasoning and retrieval from discrete language space into a continuous latent space, matching the accuracy of explicit agentic RAG while reducing inference latency by about 90%.
Background and Motivation
Single-step retrieval-augmented generation (RAG) is an efficient way to incorporate external information for simple question answering, but it struggles with complex, multi-hop questions. Agentic RAG extends the paradigm by having a large language model (LLM) act as a search agent that generates intermediate thoughts and subqueries, iteratively interacting with the retrieval system. However, this iterative process incurs substantial latency due to the autoregressive generation of lengthy natural-language thoughts and subqueries.
Core Contributions
- Latent reasoning and retrieval: Instead of generating natural-language thoughts or subqueries token by token, LatentRAG produces latent tokens for thoughts and subqueries directly from the LLM's hidden states in a single forward pass.
- Latent-space alignment: The LLM is aligned with dense retrieval models in the latent space, enabling retrieval over latent subquery tokens and supporting end-to-end joint optimization.
- Transparency via parallel latent decoding: A parallel latent decoding mechanism translates latent tokens back into natural language, improving interpretability and encouraging semantically meaningful latent representations.
- For architecture: Latent-space reasoning offers a route to keep agentic multi-step retrieval benefits without paying the autoregressive token-generation tax; retrieval strategy itself becomes a learnable component.
- For engineering: Latency, cost, and interpretability remain hard production constraints — LatentRAG's parallel decoding provides a transparency hook, but teams should still audit retrieval failure modes (empty retrievals, wrong subqueries) and validate offline gains against online metrics.
- Limitations noted by the community context: benchmark-versus-real-user distribution gaps, cross-lingual generalization, and safety of open-ended agent retrieval remain open problems.
- A Systematic Framework for Enterprise Knowledge Retrieval
- Retrieval Augmented Generation and Understanding in Vision: A Survey
- Synergizing RAG and Reasoning: A Systematic Review
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforcement Learning
- Agentic Information Retrieval
- LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG — https://arxiv.org/abs/2605.06285
Original Abstract
> Single-step retrieval-augmented generation (RAG) provides an efficient way to incorporate external information for simple question answering tasks but struggles with complex questions. Agentic RAG extends this paradigm by replacing single-step retrieval with a multi-step process, in which the large language model (LLM) acts as a search agent that generates intermediate thoughts and subqueries to iteratively interact with the retrieval system. This iterative process incurs substantial latency due to the autoregressive generation of lengthy thoughts and subqueries. To address this limitation, we propose LatentRAG, a novel framework that shifts both reasoning and retrieval from discrete language space to continuous latent space. Unlike existing explicit methods that generate natural language thoughts or subqueries token-by-token, LatentRAG produces latent tokens for thoughts and subqueries directly from the hidden states in a single forward pass. We align LLMs with dense retrieval models in the latent space, enabling retrieval over latent subquery tokens and supporting end-to-end joint optimization. To improve transparency and encourage semantically meaningful latent representations, we incorporate a parallel latent decoding mechanism that translates latent tokens back into natural language. Extensive experiments on seven benchmark datasets show that LatentRAG achieves performance comparable to explicit agentic RAG methods while reducing inference latency by approximately 90%, substantially narrowing the latency gap with traditional single-step RAG.
Evaluation
Experiments were conducted on seven benchmark datasets. LatentRAG achieves accuracy comparable to explicit agentic RAG methods while reducing inference latency by approximately 90%, substantially narrowing the latency gap with traditional single-step RAG. Exact per-dataset numbers should be verified against the original PDF.