DistMoE: Rehearsal-Free Distributed Mixture-of-Experts Routing for Multimodal Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

DistMoE is a mixture-of-experts (MoE) framework for distributed visual instruction tuning of multimodal large language models (MLLMs), proposed by Mainak Singha, Niccolò Biondi, and Elisa Ricci (arXiv:2508.03799). Adapting MLLMs to diverse visual-language domains typically assumes centralized data access, which is restrictive when data is spread across private or permission-limited clients. DistMoE augments the public feedforward network (FFN) in each language decoder layer with client-specific private FFN experts to capture domain-specific knowledge. Because independent expert training yields representations with differing scales and magnitudes, the method introduces a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mix of local and public data, with an isotropic regularization loss, enabling rehearsal-free composition across clients. At inference, DistMoE routes tokens modularly across public and private experts without explicit domain labels. Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Code is available at https://github.com/mainaksingha01/DistMoE.

Overview

Field: Computer Vision Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799

Key Points

Multimodal Large Language Models (MLLMs) show strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training — restrictive when data is distributed across private, domain-specific, or permission-limited clients.
DistMoE is a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, it augments the public feedforward network (FFN) with a client-specific private FFN expert to acquire domain-specific knowledge.
Independent expert training causes private FFNs to learn representations at different scales and magnitudes, making expert merging difficult.
To reduce client-specific drift, the authors introduce a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mixture of local client data and public data, using an isotropic regularization loss — enabling rehearsal-free composition across clients.
During inference, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without explicit domain labels.
Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while retaining modular control over client-specific knowledge.

Resources

Paper: https://arxiv.org/abs/2508.03799
Code: https://github.com/mainaksingha01/DistMoE