LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG

Authors: Yijia Zheng, Marcel Worring Source: https://arxiv.org/abs/2605.06285 Category: Agentic Search

Background and Motivation

Single-step retrieval-augmented generation (RAG) provides an efficient way to incorporate external information for simple question answering tasks but struggles with complex questions. Agentic RAG extends this paradigm by replacing single-step retrieval with a multi-step process, in which a large language model (LLM) acts as a search agent that generates intermediate thoughts and subqueries to iteratively interact with the retrieval system.

However, this iterative process incurs substantial latency due to the autoregressive generation of lengthy thoughts and subqueries.

Core Idea

LatentRAG shifts both reasoning and retrieval from discrete language space to a continuous latent space:

• Instead of generating natural-language thoughts or subqueries token-by-token, LatentRAG produces latent tokens for thoughts and subqueries directly from the LLM's hidden states in a single forward pass.
• The LLM is aligned with dense retrieval models in the latent space, enabling retrieval over latent subquery tokens and supporting end-to-end joint optimization.
• A parallel latent decoding mechanism translates latent tokens back into natural language, improving transparency and encouraging semantically meaningful latent representations.

Results

Extensive experiments on seven benchmark datasets show that LatentRAG achieves performance comparable to explicit agentic RAG methods while reducing inference latency by approximately 90%, substantially narrowing the latency gap with traditional single-step RAG.

Why It Matters

• Latency vs. quality trade-off: Agentic RAG typically trades speed for multi-hop reasoning quality; LatentRAG shows most of the reasoning can happen outside token-by-token language generation.
• Joint optimization: Aligning the LLM and retriever in a shared latent space allows the whole pipeline to be trained end-to-end rather than as separate modules.
• Interpretability: Parallel decoding of latent tokens back to text keeps the agent's intermediate steps inspectable.

Relevance

This work fits into the broader agentic search landscape alongside efforts such as enterprise knowledge retrieval frameworks, RAG-and-reasoning surveys, and reinforcement-learning-based search agents (e.g., AceSearcher). It points toward systems where reasoning, retrieval, and tool use are optimized under explicit latency and cost budgets—an important constraint for production deployments.

Limitations and Outlook

As with much of this research area, open questions include evaluation trustworthiness, cost control at scale, hallucination and safety in open-network retrieval, cross-lingual generalization, and extension to multimodal and recommendation settings. Quantitative details should be verified against the original paper's tables.

> Original abstract: "Single-step retrieval-augmented generation (RAG) provides an efficient way to incorporate external information for simple question answering tasks but struggles with complex questions. Agentic RAG extends this paradigm by replacing single-step retrieval with a multi-step process... Extensive experiments on seven benchmark datasets show that LatentRAG achieves performance comparable to explicit agentic RAG methods while reducing inference latency by approximately 90%."