Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond
Overview
This entry summarizes the paper Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond (arXiv, 2024-02-16), authored by Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua.
Motivation
Recent generative language models have shown they can memorize knowledge from documents and recall it to answer user queries. Building on this, the authors propose enabling multimodal large language models (MLLMs) to memorize and recall *images* within their parameters: given a user query for visual content, the MLLM should "recall" the relevant image from its parameters as the response. This requires inbuilt visual memory and visual recall schemes inside MLLMs.
Method
The proposed generative cross-modal retrieval framework:
1. Unique identifiers: Each image is assigned a unique identifier string. 2. Learning to memorize: Training the MLLM to memorize the association between images and their respective identifiers. 3. Learning to retrieve: Teaching the MLLM to generate the identifier of the target image given a textual query input.
By memorizing images in MLLMs, the work introduces a new generative paradigm for cross-modal retrieval, distinct from previous discriminative approaches (dual-tower dense retrieval, cross-encoder reranking, etc.).
Key Findings
- The generative paradigm performs effectively and efficiently even with large-scale image candidate sets.
- It removes the need for a separate index/ANN pipeline, shifting retrieval into model parameters.
- Quantitative results should be verified against the original paper's tables.
- Architecture: Generative retrieval eliminates cascade errors of retrieve-then-rerank pipelines, but raises index-update challenges when the image corpus changes.
- Evaluation: Beyond nDCG/MRR, generative retrieval calls for task-level success and identifier-exactness metrics.
- Deployment: Latency, cost, hallucination risks, and safe updates of memorized content are open engineering concerns.
- Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
- ColPali: Efficient Document Retrieval with Vision Language Models
- EA-VTR: Event-Aware Video-Text Retrieval (ECCV 2024)
- Hybrid-Vector Retrieval for Visually Rich Documents
Context for Search / Recommendation / Personalization
Limitations and Future Directions
Potential limitations include experiment scale, benchmark-versus-real-query distribution mismatch, and unknown cross-lingual generalization. Future work may explore more efficient test-time compute allocation, deeper integration with knowledge graphs, and causal/fairness constraints when extending to recommendation.
Related Entries
Original Abstract (verbatim)
> The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to "recall" the relevant image from its parameters as the response. Achieving this target presents notable challenges, including inbuilt visual memory and visual recall schemes within MLLMs. To address these challenges, we introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images and involves two training steps: learning to memorize and learning to retrieve. The first step focuses on training the MLLM to memorize the association between images and their respective identifiers. The latter step teaches the MLLM to generate the corresponding identifier of the target image, given the textual query input. By memorizing images in MLLMs, we introduce a new paradigm to cross-modal retrieval, distinct from previous discriminative approaches. The experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image candidate sets.