English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

DistMoE: Rehearsal-Free Expert Routing for Distributed Visual Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

DistMoE is a mixture-of-experts (MoE) approach for distributed visual instruction tuning of multimodal large language models (MLLMs), proposed by Mainak Singha, Niccolò Biondi, and Elisa Ricci (arXiv:2508.03799). Adapting MLLMs to diverse visual-language domains usually assumes centralized data access and costly joint training, which is restrictive when data is spread across private, domain-specific, or permission-limited clients. DistMoE augments the public feedforward network (FFN) in each language decoder layer with client-specific private FFN experts to capture domain-specific knowledge. Because independent expert training produces representations with differing scales and magnitudes that hinder merging, the method introduces a public-anchored expert composition stage: a router and lightweight private projection adapters are updated only on a mix of local client data and public data, with an isotropic regularization loss, enabling rehearsal-free composition across clients. At inference, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Code: https://github.com/mainaksingha01/DistMoE.

Overview

Field: CV Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799

Abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning.

In each layer of the language decoder, DistMoE augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal of acquiring domain-specific knowledge. However, independent expert training causes the private FFNs to learn representations of different scales and magnitudes, making merging the experts difficult.

To reduce client-specific drift, the authors introduce a public-anchored expert composition stage: the router and lightweight private projection adapters are updated only on a mixture of local client data and public data, with an isotropic regularization loss, enabling rehearsal-free composition across clients (i.e., without replaying or sharing raw private data).

During inference, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance, while preserving modular control over client-specific knowledge.

Resources

--- *Auto-collected on 2026-08-12.*

Tags

#mixture-of-experts#multimodal-llm#distributed-training#instruction-tuning#federated-learning#computer-vision#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633363