DistMoE: Private-data Rehearsal-free MoE Routing for Distributed Multimodal Instruction Tuning

Paper Overview

• Field: Computer Vision (CV)
• Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci
• Release date: 2026-08-11
• arXiv: 2508.03799

Summary

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, the authors propose DistMoE, a Mixture-of-Experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, DistMoE augments the public feedforward network (FFN) with a client-specific private FFN expert, aiming to acquire domain-specific knowledge.

However, independent expert training causes the private FFNs to learn representations of different scales and magnitudes, making merging the experts difficult. To reduce client-specific drift, the method introduces a publicly anchored expert composition stage that updates only the routers and lightweight private projection adapters on a mixture of local client data and public data, with an isotropic regularization loss, enabling rehearsal-free composition across clients.

At inference time, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without requiring explicit domain labels. Experiments across diverse visual-language benchmarks demonstrate flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge.

Key Points

• Problem: Centralized data access and joint training are assumed when adapting MLLMs to diverse visual-language domains, which is impractical for private or permission-limited clients.
• Method: DistMoE augments each language decoder layer's public FFN with a client-specific private FFN expert for distributed visual instruction tuning.
• Challenge addressed: Independent training of private experts produces representations with mismatched scales and magnitudes, hindering expert merging.
• Solution — Publicly anchored expert composition: Updates only routers and lightweight private projection adapters on a mix of local and public data, using an isotropic regularization loss to mitigate client drift.
• Inference behavior: Modular routing across public and private experts enables token-level domain composition without explicit domain labels.
• Results: Competitive performance on diverse visual-language benchmarks, with flexible expert reuse and modular control of client knowledge.
• Practicality: Rehearsal-free design removes the need to retain or replay client data centrally.
• Resources:
• Paper: https://arxiv.org/abs/2508.03799
• Code: https://github.com/mainaksingha01/DistMoE

Original Abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult. To reduce client...