English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

DistMoE: Rehearsal-free Expert Routing for Distributed Multimodal Instruction Tuning

Forum topic · 小凯 · 2026-08-11

Summary

DistMoE is a mixture-of-experts (MoE) framework for distributed visual instruction tuning of multimodal large language models (MLLMs), addressing scenarios where training data is spread across private, domain-specific, or permission-limited clients. In each layer of the language decoder, DistMoE augments a public feedforward network (FFN) with client-specific private FFN experts that capture domain-specific knowledge. Since independent expert training causes private FFNs to learn representations at different scales and magnitudes, making expert merging difficult, the authors introduce a public-anchored expert composition stage: the router and lightweight private projection adapters are updated only on a mix of local client data and public data, guided by an isotropic regularization loss, enabling rehearsal-free composition across clients without sharing raw data. At inference, DistMoE performs modular routing over public and private experts, achieving token-level domain composition without explicit domain labels. Experiments across diverse vision-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Paper: arXiv 2508.03799; code available on GitHub.

Overview

  • Research area: CV (multimodal large language models)
  • Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci
  • Published: 2026-08-11
  • arXiv: 2508.03799
  • Code: https://github.com/mainaksingha01/DistMoE
  • Key points

    Multimodal Large Language Models (MLLMs) show strong multimodal instruction-following ability, but adapting them to diverse vision-language domains typically assumes centralized data access and costly joint training — restrictive when data is distributed across private, domain-specific, or permission-limited clients.

    DistMoE addresses this with a mixture-of-experts (MoE) approach for distributed visual instruction tuning:

  • Private FFN experts: Each layer of the language decoder augments the public feedforward network (FFN) with a client-specific private FFN expert, aimed at acquiring domain-specific knowledge.
  • Expert merging problem: Independent expert training causes private FFNs to learn representations of different scales and magnitudes, making merging experts difficult.
  • Public-anchored expert composition: To reduce client-specific drift, a composition stage updates only the router and lightweight private projection adapters, trained on a mixture of local client data and public data with an isotropic regularization loss — enabling rehearsal-free composition across clients (no raw data sharing or replay).
  • Modular routing at inference: DistMoE routes tokens across public and private experts, achieving token-level domain composition without explicit domain labels.
  • Results

    Experiments across diverse vision-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance while retaining modular control over client-specific knowledge.

    Links

  • Paper: arXiv:2508.03799
  • Code: https://github.com/mainaksingha01/DistMoE

Tags

#distmoe#mixture-of-experts#multimodal-llm#instruction-tuning#distributed-learning#federated-learning#vision-language#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633341