MoME Explained: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition

Forum topic · QianXun · 2025-11-24

Summary

"MoME" is an acronym that can refer to multiple concepts in AI, but in the context of Meta AI research it most prominently denotes Mixture of Matryoshka Experts, a novel framework for audio-visual speech recognition (AVSR). Developed jointly by Imperial College London, Meta AI, and NatWest AI Research, MoME combines Mixture-of-Experts (MoE) sparse computation with Matryoshka Representation Learning (MRL) to address the computational inefficiency of applying large language models to multimodal data streams. The framework uses a shared router that promotes consistent expert activation across multiple input granularities, enabling implicit knowledge transfer from richer, less-compressed representations to more compressed ones. MoME achieves state-of-the-art performance on standard AVSR benchmarks (LRS2 and LRS3), as well as audio-only ASR and visual-only VSR tasks, while requiring significantly fewer active parameters during inference than competing baselines. Its dynamic capacity allocation makes it suitable for deployment across devices with varying computational budgets. The work is distinct from Meta's LLaMA series despite shared MoE principles, and a research paper describing it has been submitted to NeurIPS 2025.

Clarifying "MoME": A Guide to Multiple Meanings in AI

The acronym "MoME" appears in several AI contexts, but its most prominent recent usage refers to Mixture of Matryoshka Experts, a framework developed for audio-visual speech recognition (AVSR). This guide summarizes the framework based on the original post.

Key points

Definition: MoME (Mixture of Matryoshka Experts) synergistically combines Mixture-of-Experts (MoE) sparse computation with Matryoshka Representation Learning (MRL) to build an efficient, adaptable multimodal model that augments a pre-trained, frozen LLM.
Primary application: Audio-visual speech recognition (AVSR) — transcribing speech using both audio and lip-movement visual cues, which is especially robust in noisy environments.

How it works

The MoE component consists of multiple "expert" sub-networks plus a router (gating network) that activates only a small subset of experts per input token, drastically reducing computational cost versus dense models.
The MRL component enables the model to learn representations at multiple levels of compression simultaneously — like Russian nesting dolls — allowing "elastic inference" at different granularities without retraining.
A shared router processes tokens from all scales and modalities (audio and video), promoting consistent expert activation across granularities. This enables implicit knowledge transfer from richer, less-compressed representations to highly compressed ones, addressing performance drops typical of scale-independent MRL training.
Shared experts that are always active capture global, scale-invariant knowledge.

Technical advantages and results

Dynamic capacity allocation in two dimensions: number of active experts and input granularity, allowing adaptation to different computational budgets and accuracy requirements.
State-of-the-art performance on standard AVSR benchmarks, including the LRS2 and LRS3 datasets, as well as audio-only speech recognition (ASR) and visual-only speech recognition (VSR).
Achieves SOTA results with significantly fewer active parameters during inference than competing baselines, per the research paper.
Directly addresses the "token hunger" of LLMs, whose computational cost grows steeply with input length and granularity — a major bottleneck for continuous, high-dimensional audio-visual data streams.

Development and collaboration

Joint research by Imperial College London (including the iBUG team), Meta AI, and NatWest AI Research.
The paper, titled "MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition," has been submitted to NeurIPS 2025.
MoME is a distinct research initiative and should not be conflated with Meta's LLaMA series (e.g., LLaMA 4), even though both leverage Mixture-of-Experts architecture.

*Note: The source post was truncated; the sections above cover the complete content available.*