Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems (Google DeepMind, 2024)

Forum topic · 小凯 · 2026-07-05

Summary

This arXiv paper (2404.01616, April 2024) by researchers from Google DeepMind and the University of Edinburgh explores how decoder-only large language models (LLMs) can be transformed into efficient dual-encoder retrieval systems that operate across languages and modalities. Instead of relying on conventional encoder-based embedding models such as BERT or T5, the authors apply parameter-efficient fine-tuning (LoRA) to decoder-only LLMs, adapting them into bi-encoder retrievers suitable for large-scale search. The work investigates how such LLM-derived retrievers perform on multilingual text retrieval and cross-modal tasks, including image-text retrieval, and examines how instruction tuning and model scaling influence retrieval quality. The reported results suggest that LLM-based dual encoders can match or exceed dedicated multilingual embedding models while retaining the LLM's broad language and multimodal capabilities, making a single backbone reusable for both generation and retrieval. The paper is relevant to practitioners building multilingual search, RAG pipelines, and unified embedding systems, and offers a path toward consolidating retrieval and generation into one foundation model. Readers should consult the PDF for exact benchmark numbers.

Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems

Paper: https://arxiv.org/abs/2404.01616 Authors: Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego Affiliations: Google DeepMind, University of Edinburgh Date: April 2024 Category: Multilingual / Information Retrieval

Overview

This paper investigates how large language models, particularly decoder-only architectures, can be repurposed as retrieval systems that work across languages (cross-lingual) and modalities (cross-modal, e.g., text and images). Rather than training embedding models from scratch on encoder architectures such as BERT or T5, the authors adapt pretrained LLMs into dual-encoder (bi-encoder) retrievers using parameter-efficient fine-tuning.

Key Ideas

LLMs as retriever backbones: Decoder-only LLMs already encode strong multilingual and world knowledge; the authors show this knowledge can be leveraged for dense retrieval by adding lightweight adaptation.
Parameter-efficient adaptation: LoRA-style fine-tuning converts a generative LLM into a bi-encoder that produces query and document embeddings compatible with approximate nearest-neighbor search at scale.
Cross-lingual retrieval: The resulting retrievers are evaluated on multilingual retrieval benchmarks, testing whether LLM-derived embeddings transfer across languages without per-language tuning.
Cross-modal extension: The work examines image-text retrieval, exploring whether multimodal LLMs can serve as a single embedding backbone for both text-only and multimodal retrieval tasks.
Instruction tuning and scaling: The study considers how instruction-tuned embeddings and model scale affect retrieval quality, following trends established by instruction-tuned text embedding models.

Why It Matters

A single foundation model could serve both as a generator and as a retriever, simplifying RAG architectures and reducing the number of separately maintained models.
LLM-based dual encoders may inherit broader language coverage than traditional multilingual encoders, improving cross-lingual retrieval for low-resource languages.
Parameter-efficient fine-tuning keeps adaptation costs low compared to pretraining dedicated embedding models.

Evaluation Notes

Specific benchmark scores (e.g., nDCG@10, Recall@k) should be taken from the original paper's tables; this page summarizes the approach and scope based on the paper's public metadata.

Takeaways for Practitioners

1. Researchers: Reproduce the bi-encoder conversion and compare against dedicated multilingual embedding models on BEIR-style and MIRACL-style benchmarks. 2. Engineers: LLM-derived embeddings can potentially replace separate retrieval encoders; evaluate index size, latency, and embedding-version compatibility before migration. 3. Product teams: Unified retrieval-and-generation backbones reduce maintenance overhead, but verify latency budgets and multilingual quality on real query distributions.