DistMoE: Rehearsal-Free Expert Routing for Distributed Multimodal Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

DistMoE is a mixture-of-experts (MoE) framework for distributed visual instruction tuning of Multimodal Large Language Models (MLLMs). It augments each layer of the language decoder's public feedforward network (FFN) with a client-specific private FFN expert, enabling domain-specific adaptation without centralized data access. To address representation drift caused by independent expert training, the authors introduce a publicly anchored expert composition stage that updates only the router and lightweight private projection adapters on a mix of local and public data, using an isotropic regularization loss. This allows rehearsal-free merging of experts across clients. At inference, DistMoE performs modular token-level routing across public and private experts without explicit domain labels. Experiments on diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Code is available at the project's GitHub repository.

Overview

Field: Computer Vision (CV) Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799

Key Points

Problem: Multimodal Large Language Models (MLLMs) adapt well to visual-language domains, but traditional adaptation assumes centralized data access and expensive joint training, which is restrictive when data is distributed across private, domain-specific, or permission-limited clients.
DistMoE Framework: A mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, it augments the public feedforward network (FFN) with a client-specific private FFN expert to acquire domain-specific knowledge.
Client Drift Challenge: Independent expert training causes private FFNs to learn representations of different scales and magnitudes, making expert merging difficult.
Publicly Anchored Expert Composition: To reduce client-specific drift, the router and lightweight private projection adapters are updated only on a mix of local client data and public data. An isotropic regularization loss enables rehearsal-free expert composition across clients.
Inference Routing: At inference time, DistMoE performs modular routing across public and private experts, achieving token-level domain composition without explicit domain labels.
Results: Experiments across diverse visual-language benchmarks demonstrate flexible expert reuse, effective domain adaptation, and competitive performance, while preserving modular control over client-specific knowledge.
Code: Available at https://github.com/mainaksingha01/DistMoE

DistMoE: Rehearsal-Free Expert Routing for Distributed Multimodal Instruction Tuning

Summary

Overview

Key Points

Tags