Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems (Google DeepMind, 2024)

Paper metadata

| Field | Content | |------|------| | Title | Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems | | Authors / Affiliations | Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego (Google, DeepMind, University of Edinburgh) | | Published | April 2024 | | Link | https://arxiv.org/abs/2404.01616 | | Type | Academic paper | | Section | Multilingual |

Overview

The paper investigates how large language models can be transformed into retrieval systems that work across modalities (e.g., text and images) and across languages. It addresses core challenges in large-scale search and recommendation: efficiency, scalability, and understanding user intent. Traditional pipelines split retrieval, ranking, and generation into separate stages, which struggles to meet modern demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge.

Context in neural information retrieval

Neural IR has evolved through several generations, each balancing effectiveness, efficiency, and maintainability:

• Sparse retrieval (BM25) — robust lexical baseline;
• Dense bi-encoders — millisecond-level ANN recall, but sensitive to domain shift and long-tail queries;
• Cross-encoders — higher accuracy, but document representations cannot be precomputed;
• Late interaction, generative retrieval, and LLM agents — reduce cascade errors but complicate index updates and inference budgets.

LLM-based retrieval adds two new variables: the inference budget and the action space (whether to retrieve, how many times, and which tools to call). The paper's contribution fits into this line of adapting general-purpose LLMs into embedding models for cross-modal, cross-lingual search.

Engineering considerations

The accompanying commentary highlights deployment constraints for LLM-based retrieval systems:

• Data: PII handling, index versioning, rollback-safe embedding versions;
• Latency: p99 budgets, cascaded retrieval with early stopping, query caching, async re-ranking;
• Quality: verifying offline gains translate to online satisfaction; interleaving experiments and citation checking;
• Safety: source whitelisting, poisoning/bias detection, output filtering for open-domain retrieval;
• Cost: model routing, distillation, hybrid sparse+dense retrieval.

Related entries

• A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA
• CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents (arXiv:2504.16264)
• CL2CM: Improving Cross-lingual Cross-modal Retrieval via Cross-lingual...
• Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning (IEEE)
• Evaluating Large Language Models for Cross-Lingual Retrieval (arXiv:2509.14749)
• Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval (MM 2024, DOI: 10.1145/3664647.3680886)

Notes

This post is an editorial index entry with a structured commentary rather than a full translation of the paper. Specific numerical results should be verified against the original PDF at arXiv:2404.01616.