MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion for Video Search

Forum topic · 小凯 · 2026-07-05

Summary

MMMORRF is a video search system introduced by Saron Samuel, Dan DeGenaro, Kate Sanders, and colleagues in a March 2025 arXiv paper (2503.20698). The work addresses a key weakness in multimodal retrieval: state-of-the-art multimodal language models such as VAST and LanguageBind are built on vision-language models and over-prioritize visual signals, while benchmarks reinforce this bias by favoring visual descriptive queries. MMMORRF extracts text and features from both visual and audio modalities—capturing visual events, text overlays, sounds, and speech—and integrates them using a novel modality-aware weighted reciprocal rank fusion. The system is designed for practical information needs rather than purely visual queries. Evaluated on MultiVENT 2.0 and TVR, two multimodal benchmarks targeting targeted information needs, MMMORRF improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval, demonstrating the value of combining diverse modalities in video retrieval pipelines.

MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion

Paper: arXiv:2503.20698 Authors: Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Oluwaseun Eisape, Tanner Spendlove, et al. (16 authors) Published: 2025-03-26

Overview

Videos inherently contain multiple modalities — visual events, text overlays, sounds, and speech — all of which matter for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs) and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities.

Key contributions

MMMORRF, a search system that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion.
The system is both effective and efficient, supporting practical searches driven by users' information needs rather than purely visual descriptive queries.

Results

Evaluated on MultiVENT 2.0 and TVR — two multimodal benchmarks designed for targeted information needs:

+81% nDCG@20 over leading multimodal encoders
+37% nDCG@20 over single-modality retrieval

These results demonstrate the value of integrating diverse modalities in video retrieval.

Original abstract

> Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users' information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval, demonstrating the value of integrating diverse modalities.