MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion for Video Retrieval

Forum topic · 小凯 · 2026-07-05

Summary

MMMORRF (arXiv:2503.20698, March 2025) is a multimodal, multilingual video search system that addresses a key bias in existing multimodal retrieval models. State-of-the-art multimodal language models such as VAST and LanguageBind are built on vision-language models (VLMs), causing them to overly prioritize visual signals, while retrieval benchmarks reinforce this bias with visually descriptive queries. MMMORRF extracts text and features from both visual and audio modalities and integrates them using a novel modality-aware weighted reciprocal rank fusion. Evaluated on MultiVENT 2.0 and TVR — benchmarks targeting specific information needs rather than visual descriptions — MMMORRF improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval. The results demonstrate that combining diverse modalities substantially improves practical video search for real user information needs.

MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion

Paper: https://arxiv.org/abs/2503.20698 Published: 2025-03-26 Authors: Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Tanner Spendlove, et al. (16 authors total)

Overview

MMMORRF is a search system for videos that leverages multiple modalities — visual events, text overlays, sounds, and speech — for retrieval. The authors observe that state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs) and therefore overly prioritize visual signals. Existing retrieval benchmarks reinforce this bias by focusing on visual queries and neglecting other modalities.

Approach

MMMORRF:

1. Extracts text and features from both visual and audio modalities; 2. Integrates them with a novel modality-aware weighted reciprocal rank fusion; 3. Remains effective and efficient, supporting search based on users' actual information needs rather than visual descriptive queries.

Results

Evaluated on two multimodal benchmarks designed for more targeted information needs:

MultiVENT 2.0
TVR

Key findings:

+81% nDCG@20 over leading multimodal encoders
+37% nDCG@20 over single-modality retrieval

These results demonstrate the value of integrating diverse modalities in video retrieval.

Original Abstract

> Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users' information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval, demonstrating the value of integrating diverse modalities.

Related Work

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering (arXiv:2204.09634)
ColPali: Efficient Document Retrieval with Vision Language Models (arXiv:2407.01449)
EA-VTR: Event-Aware Video-Text Retrieval (ECCV 2024)
Generative Cross-Modal Retrieval (arXiv:2402.10805)
Hybrid-Vector Retrieval for Visually Rich Documents (arXiv:2510.22215)