DistMoE: Private-Data Rehearsal-Free Routing for Distributed Mixture-of-Experts Visual Instruction Tuning

Forum topic · 小凯 · 2026-08-12

Summary

This paper introduces DistMoE, a mixture-of-experts (MoE) framework for distributed visual instruction tuning of multimodal large language models (MLLMs) across clients with private, domain-specific, or access-restricted data. Each layer of the language decoder augments a shared public feed-forward network (FFN) with a client-specific private FFN expert, enabling domain knowledge acquisition without centralized data collection. A key challenge addressed is that independently trained private FFNs learn representations at different scales, complicating merging. The authors propose a publicly anchored expert composition stage that updates only the router and a lightweight private projection adapter on a mix of local and public data, guided by an isotropy-regularization loss, yielding rehearsal-free composition across clients. At inference, modular routing over public and private experts performs token-level domain composition without explicit domain labels. Experiments on diverse vision-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge.

Paper Overview

Research Area: Computer Vision (CV)

Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci

Published: 2026-08-12

arXiv: 2508.05146

---

Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated strong multimodal instruction-following capabilities. However, adapting them to diverse vision-language domains typically assumes centralized data access and costly joint training. This assumption is highly limiting when data is distributed across private, domain-specific, or permission-restricted clients.

To address this, the authors propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, DistMoE augments the shared public feedforward network (FFN) with a client-specific private FFN expert, aiming to acquire domain-specific knowledge.

A core challenge arises because independently trained experts cause private FFNs to learn representations of varying scales and magnitudes, which makes merging the experts difficult. To reduce client-specific drift, the authors introduce a publicly anchored expert composition stage. This stage updates only the router and a lightweight private projection adapter on a mixture of local client data and public data, guided by an isotropy regularization loss. This makes the composition rehearsal-free across clients.

At inference time, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without requiring explicit domain labels.

Key Findings

Decentralized Adaptation: DistMoE enables visual instruction tuning of MLLMs without centralized data access, making it suitable for privacy-sensitive, domain-restricted, or permission-limited clients.
Augmented FFN Design: Each decoder layer combines a shared public FFN with a client-specific private FFN expert to capture domain-specific knowledge.
Publicly Anchored Composition: A composition stage trained on local + public data mitigates scale/magnitude drift across private experts via an isotropy regularization loss, enabling rehearsal-free merging.
Lightweight Updates: Only the router and small private projection adapters are updated during composition, reducing overhead.
Domain-Label-Free Routing: Inference-time modular routing over public and private experts performs token-level domain composition without explicit domain labels.
Practical Performance: Experiments on diverse vision-language benchmarks demonstrate flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge.

Resources

Paper: arXiv:2508.05146