Paper Overview
Field: CV Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799
Abstract
Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients.
To this end, the authors propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal of acquiring domain-specific knowledge.
However, independent expert training causes the private FFNs to learn representations of different scales and magnitudes, making merging the experts difficult. To reduce client-specific drift, DistMoE introduces a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mix of local client data and public data, with an isotropic regularization loss — enabling rehearsal-free composition across clients.
During inference, DistMoE performs modular routing over the public and private experts, achieving token-level domain composition without requiring explicit domain labels.
Key Findings
- Client-specific private FFN experts augment a shared public FFN per decoder layer.
- Public-anchored composition (router + projection adapters only) removes the need for shared private-data rehearsal.
- Token-level routing works without explicit domain labels.
- Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client knowledge.