Overview
- Research area: CV (multimodal large language models)
- Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci
- Published: 2026-08-11
- arXiv: 2508.03799
- Code: https://github.com/mainaksingha01/DistMoE
- Private FFN experts: Each layer of the language decoder augments the public feedforward network (FFN) with a client-specific private FFN expert, aimed at acquiring domain-specific knowledge.
- Expert merging problem: Independent expert training causes private FFNs to learn representations of different scales and magnitudes, making merging experts difficult.
- Public-anchored expert composition: To reduce client-specific drift, a composition stage updates only the router and lightweight private projection adapters, trained on a mixture of local client data and public data with an isotropic regularization loss — enabling rehearsal-free composition across clients (no raw data sharing or replay).
- Modular routing at inference: DistMoE routes tokens across public and private experts, achieving token-level domain composition without explicit domain labels.
- Paper: arXiv:2508.03799
- Code: https://github.com/mainaksingha01/DistMoE
Key points
Multimodal Large Language Models (MLLMs) show strong multimodal instruction-following ability, but adapting them to diverse vision-language domains typically assumes centralized data access and costly joint training — restrictive when data is distributed across private, domain-specific, or permission-limited clients.
DistMoE addresses this with a mixture-of-experts (MoE) approach for distributed visual instruction tuning:
Results
Experiments across diverse vision-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance while retaining modular control over client-specific knowledge.