DistMoE: Rehearsal-Free Distributed Routing for Mixture-of-Experts in Visual Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

Adapting multimodal large language models (MLLMs) to diverse visual-language domains usually requires centralized data and expensive joint training, which is impractical when data is distributed across private, domain-specific, or permission-limited clients. The paper introduces DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, DistMoE augments the public feedforward network (FFN) with a client-specific private FFN expert to capture domain-specific knowledge. To address the challenge that independently trained private FFNs learn representations at different scales and magnitudes, the authors propose a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mix of local client data and public data, using an isotropic regularization loss. This enables rehearsal-free merging of experts across clients. At inference, DistMoE performs modular routing over public and private experts for token-level domain composition without explicit domain labels. Experiments on diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Code is released on GitHub.

Overview

Research area: Computer Vision (CV) Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799

Key Points

Problem: Multimodal Large Language Models (MLLMs) excel at multimodal instruction following, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This becomes restrictive when data is distributed across private, domain-specific, or permission-limited clients.
Proposed method (DistMoE): A mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, DistMoE augments the public feedforward network (FFN) with a client-specific private FFN expert, aiming to acquire domain-specific knowledge on each client.
Challenge addressed: Independent expert training causes private FFNs to learn representations with different scales and magnitudes, making merging experts across clients difficult.
Public-anchored expert composition: To reduce client-specific drift, the authors introduce a composition stage that updates only the router and lightweight private projection adapters on a mix of local client data and public data. An isotropic regularization loss is used to align representation magnitudes across clients, enabling rehearsal-free expert composition.
Inference: DistMoE performs modular routing over public and private experts, achieving token-level domain composition without requiring explicit domain labels.
Results: Experiments across diverse visual-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance, while preserving modular control over client-specific knowledge.
Code: https://github.com/mainaksingha01/DistMoE

Abstract (Original)

> Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult. To reduce client…

DistMoE: Rehearsal-Free Distributed Routing for Mixture-of-Experts in Visual Instruction Tuning

Summary

Overview

Key Points

Abstract (Original)

Tags