Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions (IEEE, Jan 2025)

Overview

This entry summarizes the IEEE survey "Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions" (January 2025), published on IEEE Xplore.

• Source: https://ieeexplore.ieee.org/abstract/document/10843094
• Type: Survey / Systematic Review
• Section: Search Surveys

Research Background and Motivation

In large-scale search, recommendation, and personalization systems, surveys have long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipeline-style approaches tend to separate retrieval, ranking, and generation, making it difficult to meet the demands of the LLM era: natural-language interaction, multi-hop reasoning, and real-time knowledge. This survey addresses that context, aiming to systematically map the theoretical and practical boundaries of cross-modal retrieval.

The core scenarios discussed include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation systems, and end-to-end architectures that coordinate external knowledge sources with generative models.

Core Contributions

• A unified perspective that brings dispersed related work into a comparable framework.
• A clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) to support engineering adoption.
• Reproducible benchmarks, datasets, and classification tables that lower the entry barrier for follow-up researchers.
• Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, including paths from research prototypes to industrial systems.
• Explicit open problems: evaluation credibility, latency and cost, hallucination and safety, and cross-lingual/multimodal extension.

Taxonomy

| Dimension | Sub-category | Representative ideas | Strengths | Limitations | |-----------|--------------|----------------------|-----------|-------------| | Modeling paradigm | Discriminative / generative retrieval | Dual-tower, cross-encoder, DSI, GPT-based indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Deviates from real satisfaction |

Main Research Lines Compared

The survey places four main lines of work side by side:

• Dense retrieval: high recall, low latency; suited to first-stage retrieval.
• Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
• Generative IR: directly "generates" documents via tokens or docids, simplifying the cascade.
• Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.

Timeline and Research Evolution

• 2019–2021: BERT reranking and DPR established the foundations of neural retrieval.
• 2022–2023: RAG and FreshLLM drove retrieval-generation fusion.
• 2024 onward: conversational/agentic search and Gen-RecSys exploded.
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG became new growth frontiers.

Evaluation Paradigms

Typical benchmarks and trends covered:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation sets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
• Ablations: contribution of each module (number of retrieval steps, reranking depth, training data scale).

Note: concrete numerical results should be verified against the original PDF tables; this report is based on the abstract and public metadata.

Key Conclusions and Insights

Implications for search, recommendation, and personalization:

1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms turn "retrieval count and strategy" itself into a learnable object. 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Products: latency, cost, interpretability, and safety policies are hard constraints for industrial deployment—academic benchmarks alone are not enough.

Open Problems and Future Directions

Commonly cited gaps include insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal and cross-lingual consistency.

Limitations

Potential limitations include experiment scale constrained by GPU budgets, benchmarks misaligned with real user distributions, English-centric data leaving cross-lingual generalization unknown, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.

Engineering Checklist

| Item | Question | Suggestion | |------|----------|------------| | Data | Does training/index contain PII? Version management? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? How many retrieval steps? | Cascade + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, manual audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Route to smaller models, distillation, hybrid sparse+dense |

Related Entries

• A Comprehensive Survey on Reinforcement Learning-based Agentic Search (arXiv 2510.16724)
• A Survey of Conversational Search, Oct 2024, arXiv (2410.15576)
• A Survey of LLM-Empowered Agents for Recommendation (arXiv 2503.05659)
• A Survey of Model Architectures in Information Retrieval, Jan 2025, arXiv (2502.14822)
• A Survey on AI Search with Large Language Models, July 2025
• A Survey on Knowledge-Oriented Retrieval-Augmented Generation, Mar 2025 (arXiv 2503.10677)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |