Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts Models (arXiv 2504.07859)

Forum topic · 小凯 · 2026-04-12

Summary

A new paper (arXiv:2504.07859) by Haolei Xu, Haiwen Hong, and Hongxing Li identifies a puzzling failure mode in multimodal Mixture-of-Experts (MoE) models called "Seeing but Not Thinking": the models accurately perceive image content but fail at subsequent reasoning, while correctly solving the same problems when presented as pure text. The authors verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. They further show that visual experts and domain experts are layer-wise separated, with image inputs causing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on this, they propose the Routing Distraction hypothesis: routing fails to sufficiently activate task-relevant reasoning experts when processing visual input. They introduce a routing-guided intervention to enhance domain expert activation, achieving consistent improvements across three multimodal MoE models and six benchmarks, up to 3.17% on complex visual reasoning tasks. Domain expert identification targets cognitive functions rather than sample-specific solutions, enabling transfer across tasks with different information structures.

Paper Overview

Field: NLP Authors: Haolei Xu, Haiwen Hong, Hongxing Li Published: 2025-04-10 arXiv: 2504.07859

Introduction

Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, the authors identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text.

Key Findings

Cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation for the reasoning gap.
Visual experts and domain experts exhibit layer-wise separation: image inputs induce significant routing divergence from text inputs in middle layers, where domain experts concentrate.
Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to sufficiently activate task-relevant reasoning experts.

Proposed Method

The authors design a routing-guided intervention that enhances domain expert activation during multimodal inference.

Results

Experiments across three multimodal MoE models and six benchmarks show consistent improvements.
Gains of up to 3.17% on complex visual reasoning tasks.
Domain expert identification localizes cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.

References

arXiv: 2504.07859

*Auto-collected on 2026-04-12.*