Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Research area: Multimodal learning / Mixture-of-Experts (MoE) Authors: Haolei Xu, Haiwen Hong, Hongxing Li, et al. arXiv: 2504.08290 Paper title: *Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts*

The Phenomenon

The paper describes a counterintuitive failure mode: multimodal MoE models can accurately perceive image content, yet fail at downstream reasoning on that content—while solving the same question correctly when it is presented as plain text. The problem is not "seeing" but "thinking after seeing."

MoE architectures have achieved strong results on vision-language tasks through modality-specialized experts, offering better parameter efficiency, modality specialization, and state-of-the-art benchmark performance. But this success conceals the above failure mode.

Diagnosis: Routing Distraction

Ruling out semantic misalignment

The authors first exclude the hypothesis that cross-modal semantic sharing fails: visual and textual information are indeed aligned in the model's deeper representations.

Layer-wise separation of experts

The key finding comes from analyzing expert activation patterns: vision experts and domain experts are separated across layers. Image inputs induce routing patterns in the middle layers that differ significantly from text inputs—and these middle layers are exactly where domain experts concentrate. In effect, visual inputs get routed to vision experts that monopolize middle-layer computation, while the reasoning experts that should activate for inference remain under-activated.

The Routing Distraction hypothesis

> When processing visual inputs, the routing mechanism fails to sufficiently activate task-relevant reasoning experts.

Because visual information is extremely rich (pixels, edges, textures, objects, hierarchical structure), it tends to dominate the finite routing budget, crowding out the cognitive resources needed for reasoning.

Validation: Routing-Guided Intervention

The authors design a routing-guided intervention: identify domain experts relevant to reasoning, artificially boost their routing weights when processing visual inputs, and observe the effect. No fine-tuning or retraining is involved—this is a zero-shot intervention.

Experiments across three multimodal MoE models and six benchmarks consistently show:

> Routing-guided intervention improves complex visual reasoning by up to 3.17%.

Given the zero-training setting, this gain directly supports the Routing Distraction hypothesis.

A further finding: domain experts encode cognitive functions, not sample-specific solutions. Experts specializing in mathematical reasoning, spatial relations, or causal inference transfer across tasks with different information structures.

Implications

For model design:

• Routers may need reflective capability — knowing when to suppress certain activations rather than routing feed-forward only.
• Balance expert specialization and collaboration — over-specialization risks information silos.
• Dynamic cross-modal balance — routing should adjust modality weights based on the task.

For applications:

• Visual question answering: guide models to activate reasoning experts instead of over-attending to surface image features.
• Document understanding: separate layout processing from semantic comprehension.
• Multimodal dialogue: prevent visual inputs from suppressing conversational reasoning.

Future Directions

• Move from manual intervention to learned balance: task-aware routing, adaptive gating, and meta-routing networks.
• A deeper question: if perception and representation work well but reasoning fails, should future MoE architectures include dedicated "logic experts," "causal experts," or "abstraction experts" alongside vision and language experts?

References

1. Xu, H., Hong, H., Li, H., et al. "Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts." arXiv preprint arXiv:2504.08290 (2026). 2. Fedus, W., Zoph, B., & Shazeer, N. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." JMLR 23.120 (2022): 1-39. 3. Mustafa, B., Riquelme, C., Puigcerver, J., et al. "Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts." NeurIPS 35 (2022): 14228-14241. 4. Riquelme, C., Puigcerver, J., Mustafa, B., et al. "Scaling Vision with Sparse Mixture of Experts." NeurIPS 34 (2021): 8583-8595.