Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook

Source: arXiv:2503.18016 · Survey · March 2025 · Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Lutao Jiang, Haiwei Xue, Bin Ren, et al. (10 authors) · Category: Agentic Search

One-line summary

A comprehensive survey of retrieval-augmented generation (RAG) techniques in computer vision, covering visual understanding, visual generation, and embodied AI applications, with a new outlook on future research directions.

Background and Motivation

Retrieval-augmented generation (RAG) has emerged as a pivotal AI technique, enhancing large language models (LLMs) by enabling access to external, reliable, and up-to-date knowledge sources. In AI-Generated Content (AIGC), RAG improves output quality by augmenting model outputs with relevant supplementary information. Recently, RAG has extended beyond natural language processing into computer vision (CV), addressing the limitations of relying solely on internal model knowledge by incorporating authoritative external knowledge bases.

Traditional pipelines often treat retrieval, ranking, and generation as disconnected stages, struggling to meet the LLM-era demands of natural language interaction, multi-hop reasoning, and real-time knowledge.

Core Contributions

• A unified perspective organizing dispersed RAG-for-vision work into a comparable framework
• Clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms)
• Reproducible benchmarks, datasets, and taxonomy tables to lower the entry barrier for researchers
• Discussion of interfaces with LLM tool use, reinforcement learning, and multi-agent collaboration, plus migration paths from research prototypes to industrial systems
• Explicit open problems: evaluation reliability, latency/cost, hallucination and safety, cross-lingual and multimodal scaling

Scope of the Survey

• Visual understanding: tasks from basic image recognition to complex applications such as medical report generation and multimodal question answering
• Visual generation: RAG applied to image, video, and 3D generation
• Embodied AI: RAG in planning, task execution, multimodal perception, interaction, and specialized domains

Taxonomy of Retrieval Approaches

| Dimension | Subtypes | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Dual-tower, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization goals | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Annotation scarcity | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B testing | Comparable | Deviates from real satisfaction |

Four main research lines are typically contrasted:

• Dense retrieval: high recall, low latency, suited to first-stage retrieval
• Late interaction (e.g., ColBERT): higher accuracy but larger index
• Generative IR: directly "generates" documents via tokens or docids, simplifying cascades
• Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection

Timeline of Research Evolution

• 2019–2021: BERT reranking and DPR establish neural retrieval foundations
• 2022–2023: RAG and FreshLLM drive retrieval-generation fusion
• 2024 onward: explosive growth in conversational/agentic search and Gen-RecSys
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth frontiers

Evaluation Paradigms

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, recommendation benchmarks
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs
• Ablations: contributions of retrieval steps, reranking depth, and training data scale

Specific numerical results should be verified against the original PDF.

Key Insights

1. Architecture: cascaded retrieve–rerank–generate remains mainstream, but the agentic paradigm makes retrieval count and strategy themselves learnable 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data requires protection against knowledge leakage and distribution shift 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation 4. Product: latency, cost, interpretability, and safety are hard constraints for industrial deployment—not just academic benchmark optimization

Limitations and Future Directions

Noted gaps include insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, retrieval-reasoning co-training, enterprise metadata governance, and multimodal/cross-lingual consistency. The authors also flag GPU-budget constraints, benchmark–real-user distribution mismatch, English-centric data, and security risks of agents operating on the open web.

Original Abstract (excerpt)

> Retrieval-augmented generation (RAG) has emerged as a pivotal technique in artificial intelligence (AI), particularly in enhancing the capabilities of large language models (LLMs) by enabling access to external, reliable, and up-to-date knowledge sources. … This survey provides a comprehensive review of the current state of retrieval-augmented techniques in CV, focusing on two main areas: (I) visual understanding and (II) visual generation. … Given that the integration of retrieval-augmented techniques in CV is still in its early stages, we also highlight the key limitations of current approaches and propose future research directions to drive the development of this promising area.

Related Reading

• Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
• Agentic Information Retrieval (arXiv:2410.09713)
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via RL

Glossary

| Term | Meaning | |---|---| | RAG | Retrieval-Augmented Generation | | IR | Information Retrieval | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Gen-IR | Generative Information Retrieval | | Agentic Search | Modeling search as sequential decision-making and tool use by an agent |