DistMoE: Rehearsal-Free Mixture-of-Experts Routing for Distributed Visual Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

DistMoE is a mixture-of-experts (MoE) framework for adapting multimodal large language models to diverse visual-language domains when training data is distributed across private or permission-limited clients. Instead of centralized joint training, DistMoE augments the public feedforward network (FFN) in each language decoder layer with client-specific private FFN experts that capture domain-specific knowledge. Because independent expert training produces representations at differing scales and magnitudes, making expert merging difficult, the method introduces a public-anchored expert composition stage: the router and lightweight private projection adapters are updated on a mixture of local client data and public data with an isotropic regularization loss, enabling rehearsal-free composition across clients. At inference, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while retaining modular control over client-specific knowledge. Paper: arXiv 2508.03799; code available on GitHub.

Paper Overview

Field: Computer Vision Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799

Abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To address this, the authors propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning.

Key Ideas

Private FFN experts: In each layer of the language decoder, the public feedforward network (FFN) is augmented with a client-specific private FFN expert, aimed at acquiring domain-specific knowledge.
Expert merging challenge: Independent expert training causes private FFNs to learn representations with different scales and magnitudes, making it difficult to merge experts.
Public-anchored composition: To reduce client-specific drift, DistMoE introduces a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mixture of local client data and public data, using an isotropic regularization loss. This enables rehearsal-free composition across clients without accessing raw private data from others.
Token-level routing at inference: DistMoE performs modular routing over public and private experts, achieving token-level domain composition without requiring explicit domain labels.

Results

Experiments across diverse visual-language benchmarks show that DistMoE achieves flexible expert reuse, effective domain adaptation, and competitive performance, while preserving modular control over client-specific knowledge.

Resources

Paper: arXiv:2508.03799
Code: https://github.com/mainaksingha01/DistMoE