Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models

• Paper: Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond
• Authors: Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, Tat-Seng Chua
• Published: 2024-02-16 (arXiv)

Key idea

Generative language models can memorize knowledge from documents and recall it to answer queries. This paper extends that capability to images: a multimodal large language model (MLLM) memorizes images within its own parameters, so that given a textual query for visual content, it "recalls" the relevant image by generating its identifier — a generative alternative to discriminative cross-modal retrieval.

Method

1. Identifier assignment — each image is represented by a unique identifier string. 2. Learning to memorize — the MLLM is trained to associate images with their identifiers. 3. Learning to retrieve — the MLLM learns to generate the identifier of the target image given a textual query.

This replaces the conventional retrieve-and-rerank pipeline (dense retrievers, cross-encoder rerankers) with a single generative step.

Results and significance

Experiments show the generative paradigm is effective and efficient even with large-scale image candidate sets. It introduces a new paradigm for cross-modal retrieval, distinct from previous discriminative approaches.

Discussion

As a generative information retrieval (Gen-IR) approach, it fits the broader shift from BM25 → dense retrieval → late interaction → generative retrieval and LLM-based agents. Open engineering questions include index updates (adding/removing images requires retraining), retrieval latency and cost, hallucination of wrong identifiers, and evaluation beyond offline metrics such as Recall@k and nDCG@10.

Original abstract

> The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to "recall" the relevant image from its parameters as the response. Achieving this target presents notable challenges, including inbuilt visual memory and visual recall schemes within MLLMs. To address these challenges, we introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images and involves two training steps: learning to memorize and learning to retrieve. The first step focuses on training the MLLM to memorize the association between images and their respective identifiers. The latter step teaches the MLLM to generate the corresponding identifier of the target image, given the textual query input. By memorizing images in MLLMs, we introduce a new paradigm to cross-modal retrieval, distinct from previous discriminative approaches. The experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image candidate sets.