Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning (IEEE 2024)

Overview

This entry indexes an IEEE 2024 paper on cross-lingual cross-modal retrieval with noise-robust fine-tuning.

| Field | Value | |-------|-------| | Title | Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning | | Venue | IEEE, 2024 | | Source | IEEE Xplore | | Category | Multi Lingual |

Problem and Motivation

Cross-lingual cross-modal retrieval aims to retrieve images (or other media) whose captions are in a different language from the query. Because large-scale multilingual image-text pairs are rarely clean — pairs are often mined from the web or constructed via machine translation — training data contains significant alignment and label noise. Standard fine-tuning on such data degrades retrieval quality, especially for the target language.

Core Idea

The paper proposes a noise-robust fine-tuning strategy that lets a pretrained vision-language model absorb cross-lingual alignment despite noisy supervision. The method belongs to the broader line of work on adapting contrastively pretrained models (e.g., CLIP-style encoders) to new languages without retraining from scratch on clean multilingual data.

Context in the Literature

The paper connects to several threads in neural information retrieval:

• Dense retrieval and dual-tower encoders: efficient ANN-based recall, but sensitive to domain and language shift.
• Vision-language pretraining: CLIP-style models are strong on English but degrade cross-lingually.
• Noisy-label learning: robustness techniques adapted here to the multimodal multilingual setting.

Related entries in this list:

• CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents
• CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Perturbation
• Evaluating Large Language Models for Cross-Lingual Retrieval (Sep 2025)
• Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval (MM 2024)
• The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora

Takeaways for Practitioners

1. Data noise is the bottleneck for extending multimodal retrievers to new languages; robust fine-tuning matters more than raw data volume. 2. Evaluate cross-lingually: offline metrics computed on English test sets do not predict multilingual retrieval quality. 3. Check the original paper for exact datasets, metrics (Recall@k, MRR), and ablations — the quantitative results are not reproduced here.

Glossary

| Term | Meaning | |------|---------| | CLIR | Cross-Lingual Information Retrieval | | CLCMR | Cross-Lingual Cross-Modal Retrieval | | nDCG | Normalized Discounted Cumulative Gain | | RAG | Retrieval-Augmented Generation |

> Original abstract (as archived): "Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning, IEEE 2024"

See the IEEE Xplore page for the full paper.