English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

DistMoE: Rehearsal-Free MoE Routing for Distributed Multimodal Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

DistMoE is a mixture-of-experts (MoE) approach for distributed visual instruction tuning of multimodal large language models (MLLMs), addressing scenarios where training data is spread across private, domain-specific, or permission-limited clients. Presented by Mainak Singha, Niccolò Biondi, and Elisa Ricci (arXiv:2508.03799), the method augments the public feedforward network (FFN) in each layer of the language decoder with client-specific private FFN experts that capture domain-specific knowledge. Because independent expert training produces representations at differing scales and magnitudes, merging experts is difficult; DistMoE introduces a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mix of local client and public data, using an isotropic regularization loss to reduce client drift without rehearsing raw private data. At inference, modular routing over public and private experts enables token-level domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while retaining modular control over client-specific knowledge. Code is available at https://github.com/mainaksingha01/DistMoE.

Paper Overview

Field: CV Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-11 arXiv: 2508.03799

Abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients.

To this end, the authors propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal of acquiring domain-specific knowledge.

However, independent expert training causes the private FFNs to learn representations of different scales and magnitudes, making merging the experts difficult. To reduce client-specific drift, DistMoE introduces a public-anchored expert composition stage that updates only the router and lightweight private projection adapters on a mix of local client data and public data, with an isotropic regularization loss — enabling rehearsal-free composition across clients.

During inference, DistMoE performs modular routing over the public and private experts, achieving token-level domain composition without requiring explicit domain labels.

Key Findings

  • Client-specific private FFN experts augment a shared public FFN per decoder layer.
  • Public-anchored composition (router + projection adapters only) removes the need for shared private-data rehearsal.
  • Token-level routing works without explicit domain labels.
  • Experiments across diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client knowledge.
Code: https://github.com/mainaksingha01/DistMoE

Tags

#mixture-of-experts#multimodal-llm#instruction-tuning#distributed-learning#federated-learning#visual-language-models#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633353