Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Forum topic · 小凯 · 2026-04-11

Summary

This paper (arXiv:2504.07076) identifies a puzzling failure mode in multimodal Mixture-of-Experts (MoE) models called "Seeing but Not Thinking": models accurately perceive image content yet fail to reason over it, while correctly solving the same problems presented as plain text. The authors verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic misalignment as the sole cause. They further show that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing routing divergence from text inputs in middle layers where domain experts concentrate. From this they propose the Routing Distraction hypothesis: when processing visual inputs, the router fails to adequately activate task-relevant reasoning experts. A routing-guided intervention that enhances domain expert activation yields consistent improvements across three multimodal MoE models and six benchmarks, with gains up to 3.17% on complex visual reasoning. Analysis shows domain experts encode cognitive functions rather than sample-specific solutions, enabling transfer across tasks.

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Field: AI Authors: Haolei Xu, Haiwen Hong, Hongxing Li Published: 2025-04-10 arXiv: 2504.07076

Abstract

Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text.

Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate.

Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts.

To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.

Key findings

Seeing but Not Thinking phenomenon: multimodal MoE models perceive images correctly but fail to reason over them, succeeding on the same questions given as text.
Cross-modal semantic sharing exists in MoE architectures, so semantic misalignment is not the sole cause.
Visual experts vs. domain experts are layer-wise separated; image inputs cause routing divergence from text in middle layers where domain experts concentrate.
Routing-guided intervention enhancing domain expert activation brings gains up to 3.17% on complex visual reasoning across three models and six benchmarks.
Domain experts encode cognitive functions, not sample-specific solutions, allowing transfer across differently structured tasks.

---

*Auto-collected on 2025-04-11.*

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Summary

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Abstract

Key findings

Tags