English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

DistMoE: Rehearsal-Free Distributed Mixture-of-Experts Routing for Visual Instruction Tuning

Forum topic · 小凯 · 2026-08-12

Summary

DistMoE (arXiv 2508.05146) is a mixture-of-experts approach for distributed visual instruction tuning of multimodal large language models (MLLMs). Addressing the limitation that adapting MLLMs typically requires centralized data access, DistMoE supports clients with private, domain-specific, or permission-limited data. Each layer of the language decoder augments a public feedforward network (FFN) with a client-specific private FFN expert to capture domain knowledge. Because independent expert training produces representations at different scales, making merging difficult, the authors introduce a public anchored expert combination stage: the router and lightweight private projection adapters are updated only on a mix of local client data and public data, regularized with an isotropic loss to reduce client-specific drift, eliminating the need for data rehearsal. At inference, DistMoE performs modular routing across public and private experts, achieving per-token domain composition without explicit domain labels. Experiments on diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge.

Paper Overview

Field: Computer Vision Authors: Mainak Singha, Niccolò Biondi, Elisa Ricci Published: 2026-08-12 arXiv: 2508.05146

Abstract (Translation)

Multimodal Large Language Models (MLLMs) show strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training — restrictive when data is distributed across private, domain-specific, or permission-limited clients. DistMoE is a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder, it augments the public feedforward network (FFN) with a client-specific private FFN expert to acquire domain-specific knowledge. However, independent expert training causes private FFNs to learn representations of different scales and magnitudes, making expert merging difficult. To reduce client-specific drift, DistMoE introduces a public anchored expert combination stage in which only the router and lightweight private projection adapters are updated on a mixture of local client data and public data, enforced via an isotropic regularization loss — making it rehearsal-free across clients. At inference, DistMoE performs modular routing over public and private experts, enabling per-token domain composition without explicit domain labels. Experiments on diverse visual-language benchmarks show flexible expert reuse, effective domain adaptation, and competitive performance while retaining modular control over client-specific knowledge.

Original Abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult...

---

*Auto-collected on 2026-08-12*

Tags

#mixture-of-experts#multimodal-llm#distributed-learning#visual-instruction-tuning#federated-learning#computer-vision#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633375