Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook

Source: https://arxiv.org/abs/2503.18016 · Posted 2025-03-23 · Survey · Authors: Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Lutao Jiang, Haiwei Xue, Bin Ren, et al. (10 authors)

Original Abstract

> Retrieval-augmented generation (RAG) has emerged as a pivotal technique in artificial intelligence (AI), particularly in enhancing the capabilities of large language models (LLMs) by enabling access to external, reliable, and up-to-date knowledge sources. In the context of AI-Generated Content (AIGC), RAG has proven invaluable by augmenting model outputs with supplementary, relevant information, thus improving their quality. Recently, the potential of RAG has extended beyond natural language processing, with emerging methods integrating retrieval-augmented strategies into the computer vision (CV) domain. These approaches aim to address the limitations of relying solely on internal model knowledge by incorporating authoritative external knowledge bases, thereby improving both the understanding and generation capabilities of vision models. This survey provides a comprehensive review of the current state of retrieval-augmented techniques in CV, focusing on two main areas: (I) visual understanding and (II) visual generation. In the realm of visual understanding, we systematically review tasks ranging from basic image recognition to complex applications such as medical report generation and multimodal question answering. For visual content generation, we examine the application of RAG in tasks related to image, video, and 3D generation. Furthermore, we explore recent advancements in RAG for embodied AI, with a particular focus on applications in planning, task execution, multimodal perception, interaction, and specialized domains. Given that the integration of retrieval-augmented techniques in CV is still in its early stages, we also highlight the key limitations of current approaches and propose future research directions to drive the development of this promising area.

Key Points

• Scope: Comprehensive review of retrieval-augmented techniques across visual understanding and visual generation, plus embodied AI applications.
• Visual understanding: Covers tasks from basic image recognition to medical report generation and multimodal question answering.
• Visual generation: Examines RAG in image, video, and 3D generation pipelines.
• Embodied AI: Discusses retrieval-augmented planning, task execution, multimodal perception, and interaction.
• Positioning: The integration of RAG into computer vision is still early-stage; the survey identifies key limitations and future research directions.

Taxonomy and Research Context

The survey situates itself at the intersection of RAG and large-scale search/recommendation systems. Related work is commonly organized along four main lines:

• Dense retrieval: high recall, low latency; suited for first-stage retrieval.
• Late interaction (e.g., ColBERT): higher precision but larger index footprint.
• Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
• Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.

A typical taxonomy spans:

| Dimension | Subcategories | Trade-offs | |---|---|---| | Modeling paradigm | Discriminative vs. generative retrieval (dual-encoder, cross-encoder, DSI) | Mature and scalable vs. semantic drift, update cost | | LLM integration | RAG / agents / tool use | Flexible and interpretable vs. latency, error propagation | | Optimization goals | Relevance, diversity, freshness | Business alignment vs. label scarcity | | Evaluation | Offline / online / human (nDCG, MRR, LLM-as-judge) | Comparable vs. gap with real user satisfaction |

Historically, BERT reranking and DPR (2019–2021) established neural retrieval; RAG and FreshLLM (2022–2023) drove retrieval-generation fusion; conversational/agentic search and Gen-RecSys surged from 2024; and RL-trained search agents, Deep Research, and GraphRAG are emerging growth areas for 2025–2026.

Evaluation Landscape

• Datasets: MS MARCO, BEIR, Natural Questions, domain corpora.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, task success rate, latency, token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
• Trend: Evaluation is shifting from static nDCG toward process-level metrics such as task success rate, citation accuracy, and multi-hop reasoning chain completeness.

Insights and Open Problems

1. Architecture: Cascaded retrieve–rerank–generate remains mainstream, but agentic paradigms make retrieval count and strategy themselves learnable. 2. Data: High-quality instruction data and click/session logs are critical; synthetic data risks knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Deployment: Latency, cost, interpretability, and safety are hard industrial constraints. 5. Open questions: Unified benchmarks, reproducibility with private data, LLM evaluation bias, and safety/cost constraints of agentic systems; future work includes finer-grained process supervision, retrieval-reasoning co-training, deeper integration with knowledge graphs, and multimodal/cross-lingual consistency.

Related Entries

• Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
• Agentic Information Retrieval (arXiv:2410.09713)
• AceSearcher: Bootstrapping Reasoning and Search for LLMs (arXiv:2509.24193)

> Note: Quantitative details should be verified against the original PDF; this post is organized from the abstract and public metadata.