Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook

Paper: arXiv:2503.18016 · Survey · Published 2025-03-23 Authors: Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Lutao Jiang, Haiwei Xue, Bin Ren, et al. (10 authors total)

Abstract (original)

> Retrieval-augmented generation (RAG) has emerged as a pivotal technique in artificial intelligence (AI), particularly in enhancing the capabilities of large language models (LLMs) by enabling access to external, reliable, and up-to-date knowledge sources. In the context of AI-Generated Content (AIGC), RAG has proven invaluable by augmenting model outputs with supplementary, relevant information, thus improving their quality. Recently, the potential of RAG has extended beyond natural language processing, with emerging methods integrating retrieval-augmented strategies into the computer vision (CV) domain... This survey provides a comprehensive review of the current state of retrieval-augmented techniques in CV, focusing on two main areas: (I) visual understanding and (II) visual generation... we also highlight the key limitations of current approaches and propose future research directions.

Key points

• Scope: A systematic survey of retrieval-augmented techniques across computer vision, covering both visual understanding (image recognition through medical report generation, multimodal question answering) and visual generation (image, video, and 3D generation).
• Embodied AI coverage: Reviews RAG advances in embodied agents — planning, task execution, multimodal perception, interaction, and specialized domains.
• Motivation: RAG addresses the limits of relying solely on a model's internal knowledge by incorporating authoritative external knowledge bases, improving both understanding and generation quality.

A taxonomy of retrieval-augmented methods

| Dimension | Sub-classes | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Dual-tower, cross-encoder, DSI | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization targets | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Label scarcity | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B | Comparable | Gap vs. real satisfaction |

Four main research lines are commonly contrasted:

• Dense retrieval — high recall, low latency; suited to first-stage retrieval.
• Late interaction (e.g., ColBERT) — higher precision but larger indexes.
• Generative IR — "generates" documents via tokens/docids, simplifying cascades.
• Agentic search — models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.

Evolution timeline

• 2019–2021: BERT re-ranking and DPR establish neural retrieval.
• 2022–2023: RAG and FreshLLM drive retrieval–generation fusion.
• 2024 onward: Conversational/agentic search and Gen-RecSys surge.
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG emerge as growth frontiers.

Insights and open problems

1. Architecture: Cascaded retrieve → rerank → generate remains dominant, but agentic paradigms make retrieval strategy itself learnable. 2. Data: High-quality instruction data and click/session logs matter equally; synthetic data risks leakage and distribution shift. 3. Evaluation: Offline metrics diverge from online satisfaction; LLM-as-judge needs human cross-validation. 4. Product constraints: Latency, cost, interpretability, and safety are hard industrial requirements, not just benchmark numbers.

Open problems highlighted include: lack of unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, retrieval–reasoning co-training, enterprise metadata governance, and multimodal/cross-lingual consistency.

Related entries

• Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforcement Learning (arXiv:2509.24193)
• Agentic Information Retrieval (arXiv:2410.09713)

Glossary

| Term | Meaning | |---|---| | RAG | Retrieval-Augmented Generation | | IR | Information Retrieval | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Gen-IR | Generative Information Retrieval | | Agentic Search | Search modeled as sequential decision-making and tool use |