Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts Models
*This post is a structured English summary of a Chinese forum article discussing a research paper on multimodal MoE models.*
Key points
- The phenomenon: Researchers from Zhejiang University and Alibaba describe "Seeing but Not Thinking" — multimodal MoE models correctly perceive visual content but fail to reason over it. Example: Qwen3-VL-30B-A3B solves a simple math problem in text form, but answers incorrectly when the identical problem is rendered as a high-resolution image, despite accurately recognizing every digit and character in it.
- Error attribution: Using MATH500 problems rendered as images, the study finds 68.2%–73.1% of failures are reasoning errors, while only 26.9%–31.8% are perception errors — the model usually "sees" correctly but "thinks" incorrectly.
- Background on MoE: Mixture-of-Experts models use a router that selectively activates a few specialized experts per input (e.g., vision experts, math/reasoning experts, language experts), enabling scalability and specialization. This architecture powers Qwen3-VL, Kimi-VL, and Llama4.
- Routing Distraction Hypothesis:
- Vision experts cluster in the bottom layers (raw visual features) and top layers (modality-specific output), while domain (reasoning) experts concentrate in the middle layers.
- Visual input causes routing divergence in middle layers (6–42): the router fails to activate reasoning experts even when content is correctly recognized. Larger Jensen-Shannon divergence between image and text routing correlates with lower accuracy.
- Semantic misalignment ruled out: A cross-modal concept intervention experiment (swapping hidden-state vectors of digits between image and text inputs) shows an inverted-U pattern: intervention success exceeds 90% in middle layers (8–42), indicating strong cross-modal semantic sharing there.
- Perception errors ruled out: Manual error analysis confirms most failures are reasoning, not perception, errors.
- Soft intervention: boost identified domain experts' router logits by
λ · s(r_l)(scaled by layer logit std-dev). - Hard intervention: set domain experts' logits to the layer maximum:
r'_{l,k} ← max_j(r_{l,j}) + δ. - Random baseline: boosting randomly chosen experts, to verify gains come from activating the *right* experts.
- Models tested: Qwen3-VL-30B-A3B, Kimi-VL-16B-A3B, Llama4-Scout-109B-A17B.
- Benchmarks: MATH500, GPQA-Diamond, MathVerse, MATH-Vision, GSM8K-V.
- Soft intervention: +3.17% on MATH500 (Qwen3-VL), +2.73% on GPQA-Diamond physics (Kimi-VL); random baseline shows negligible effect.
- Hard intervention works better for naturally visual tasks (geometry, function plots) that text cannot replace.
- Identified experts transfer across sample formats, suggesting experts encode cognitive functions (e.g., "math reasoning") rather than sample-specific memorization.
- Xu, H., et al. (2026). *Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts*. arXiv:2604.08541.
- Bai, S., et al. (2025). *Qwen3-VL Technical Report*. arXiv:2511.21631.
- Team, K., et al. (2025). *Kimi-VL Technical Report*. arXiv:2504.07491.
- Hendrycks, D., et al. (2021). *Measuring Mathematical Problem Solving With the MATH Dataset*. NeurIPS.
Ruling out alternatives
Proposed fix: inference-time routing intervention
Results
Limitations noted by the authors
1. Routing intervention cannot fix perception errors. 2. Expert identification and intervention hyperparameters require task-specific tuning. 3. Cross-modal semantic alignment of complex concepts (spatial relations, geometric configurations) remains unverified. 4. Correlation between routing divergence and accuracy does not fully establish causality.
Takeaway
The study reframes multimodal reasoning failures in MoE models as an attention-allocation problem: visual input distracts the router from reasoning experts. Simple inference-time guidance — nudging the router toward the right experts — recovers part of the lost accuracy without retraining or architectural changes.