Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions (IEEE, Jan 2025)

Forum topic · 小凯 · 2026-07-05

Summary

A community discussion summarizing the IEEE survey "Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions" (January 2025), published on IEEE Xplore. The post situates the survey within the broader evolution of neural information retrieval: from BM25 and BERT-based reranking, through dense retrieval (dual encoders, DPR), late interaction models such as ColBERT, generative retrieval (DSI), to retrieval-augmented generation (RAG) and agentic search. It organizes methods along dimensions including modeling paradigm, LLM integration, optimization objectives, and evaluation protocols, and contrasts the four major research lines (dense, late interaction, generative IR, agentic search) on recall, latency, accuracy, and index cost trade-offs. The discussion also covers benchmark datasets (MS MARCO, BEIR, Natural Questions), metrics (nDCG@10, MRR, Recall@k, LLM-as-judge), and open problems such as unified benchmarks, evaluation bias in LLM judges, hallucination and safety, multilingual and multimodal generalization, and the engineering constraints of latency, cost, and explainability when deploying search and recommendation systems in the LLM era. Quantitative claims should be verified against the original PDF.

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions (IEEE, Jan 2025)

Source: IEEE Xplore. Type: Survey, cataloged under "Search Surveys".

Overview

This forum post summarizes the IEEE survey on cross-modal retrieval, framing it within the LLM-era rethinking of how retrieval, ranking, generation, and tool invocation should divide responsibilities. The survey provides a unified, comparable taxonomy across a fragmented literature and identifies interfaces with LLM tool use, reinforcement learning, and multi-agent collaboration.

Taxonomy of Methods

| Dimension | Subcategories | Representative Ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Dual encoders, cross-encoders, DSI, GPT-indexed documents | Mature, scalable | Semantic drift, index update cost | | LLM integration | RAG / agents / tool use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce supervision | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Diverges from real user satisfaction |

Four Research Lines Compared

Dense retrieval: high recall, low latency; suited for first-stage retrieval.
Late interaction (e.g., ColBERT): higher accuracy but larger index footprint.
Generative IR: directly "generates" documents via tokens or docids, simplifying the cascade.
Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.

Timeline of the Field

2019–2021: BERT rerankers and DPR establish neural retrieval foundations.
2022–2023: RAG and FreshLLM drive retrieval-generation integration.
2024 onward: conversational/agentic search and Gen-RecSys surge.
2025–2026: RL-trained search agents, Deep Research, and GraphRAG emerge as growth areas.

Evaluation Practices

Typical datasets: MS MARCO, BEIR, Natural Questions, and domain corpora. Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost. Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, and commercial search APIs. Specific numbers should be verified against the original paper's tables.

Key Insights

1. Architecture: cascade retrieve-rerank-generate remains mainstream, but agentic paradigms make retrieval strategy itself learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data risks leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Deployment: latency, cost, explainability, and safety are hard industrial constraints, not just benchmark optimization.

Open Problems

Limited unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints for agentic systems. Future work: finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, multimodal and cross-lingual consistency, deeper integration with knowledge graphs, and causal/fairness constraints for recommendation.

Engineering Checklist

| Item | Question | Recommendation | |---|---|---| | Data | PII in index/training? Version control? | Partitioned indexes, sanitization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade with early exit, query caching, async reranking | | Quality | Do offline gains translate online? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query tokens / GPU usage? | Small-model routing, distillation, hybrid sparse-dense retrieval |

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions (IEEE, Jan 2025)

Summary

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions (IEEE, Jan 2025)

Overview

Taxonomy of Methods

Four Research Lines Compared

Timeline of the Field

Evaluation Practices

Key Insights

Open Problems

Engineering Checklist

Related Items

Tags