Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval (ACM MM 2024)

Overview

This entry indexes the academic paper "Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval", published at ACM Multimedia 2024.

• Venue: ACM MM 2024
• DOI / Source: https://dl.acm.org/doi/abs/10.1145/3664647.3680886
• Category: Multilingual information retrieval / cross-modal retrieval

Problem Setting

Cross-lingual cross-modal retrieval requires matching content across two axes of mismatch:

1. Language: queries and targets are written in different languages (e.g., a Chinese text query retrieving English-described images). 2. Modality: matching spans text, images, and other media types.

Traditional CLIP-style dual-encoder pipelines often struggle with low-resource languages and fine-grained cross-modal alignment. This work explores how multimodal large language models (MLLMs) — with their strong vision-language grounding and multilingual pretraining — can be used to enhance retrieval representations and generalization.

Position in the Literature

The paper sits within a rapidly evolving line of work on multilingual multimodal retrieval. Related entries in this collection include:

• CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents (arXiv:2504.16264)
• Cl2cm: Improving Cross-lingual Cross-modal Retrieval via Cross-lingual ...
• Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning (IEEE)
• Evaluating Large Language Models for Cross-Lingual Retrieval (arXiv:2509.14749)
• The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora (arXiv:2507.07543)

Broader Context: LLM-Era Retrieval

The surrounding discussion frames the paper within the general shift in information retrieval:

• Architecture: cascade retrieval + re-ranking + generation remains dominant, but agentic paradigms increasingly treat "when and how many times to retrieve" as a learnable decision.
• Data: high-quality instruction data and click/session logs remain critical; synthetic data must guard against leakage and distribution shift.
• Evaluation: the gap between offline metrics (nDCG, Recall@k) and online user satisfaction is widening; LLM-as-judge should be cross-validated with human assessment.
• Deployment: latency, cost, interpretability, and safety are hard constraints for production systems, beyond academic benchmarks.

Open Challenges

• Trustworthy evaluation benchmarks for cross-lingual settings
• Latency and inference cost of MLLM-enhanced retrievers
• Hallucination and safety in generative retrieval components
• Generalization beyond English-centric training data

Caveat

This post is primarily an index entry with templated commentary. For the exact model architecture, training recipe, datasets, and quantitative results, please consult the original paper at the ACM Digital Library link above.