Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval (ACM MM 2024)
Metadata
| Field | Content | |-------|---------| | Title | Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval, MM 2024 | | Venue | ACM Multimedia 2024 | | Source | https://dl.acm.org/doi/abs/10.1145/3664647.3680886 | | Resource type | Academic paper | | Section | Multi Lingual |
Overview
This ACM Multimedia 2024 paper targets cross-lingual cross-modal retrieval — retrieving visual content (images/videos) with queries in a different language, or text in one language matching captions in another — enhanced with multimodal large language models (LLMs). Note: detailed experimental figures are not reproduced here; consult the original PDF via the DOI link above.
Background and Motivation
In large-scale search, recommendation, and personalization systems, information retrieval has long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipelines treat retrieval, ranking, and generation as separate stages, which is ill-suited to the LLM era's demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. Cross-lingual cross-modal retrieval is a particularly hard instance: models must align semantics across two dimensions simultaneously — language and modality — where annotated paired data is scarce and domain transfer is difficult. Multimodal LLMs offer rich semantic priors that can bridge these gaps.
Method: Typical Architecture
Works in this space generally follow a four-step recipe: problem formalization → model/system design → training/construction → inference pipeline.
1. Input & representation: encode queries, documents, and visual content into dense or sparse representations, or structured prompts; 2. Core modules: retrievers, rerankers, planners, memory modules, tool interfaces, composed in series or parallel; 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (incl. process rewards), bootstrapped data synthesis; 4. Inference strategies: single-shot retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Evaluation (general protocol in this area)
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, multilingual/multimodal benchmarks;
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost;
- Baselines: BM25, dense retrieval, cross-encoder rerankers, retrieval-free LLMs, commercial search APIs;
- Ablations: contribution of each module (retrieval steps, rerank depth, training data scale).
- A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA
- CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents
- Cl2cm: Improving cross-lingual cross-modal retrieval via cross-lingual...
- Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning
- Evaluating Large Language Models for Cross-Lingual Retrieval, Sep 2025
- The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora
- Original paper: *Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval*, ACM MM 2024 — https://dl.acm.org/doi/abs/10.1145/3664647.3680886
Specific numbers must be verified against the original paper's tables.
Key Takeaways for Search / Rec / Personalization
1. Architecture: cascade retrieval + rerank + generation remains mainstream, but agentic paradigms make the *retrieval policy itself* learnable; 2. Data: high-quality instruction data and click/session logs are as critical as architectures; synthetic data must avoid leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation; 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment — do not optimize only for academic benchmarks.
Limitations and Future Directions
Possible limitations include experiment scale constrained by GPU budgets, benchmark–real-user distribution mismatch, English-centric data limiting cross-lingual generalization, and safety risks of agentic systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.
Cross-references
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |