The Routing Distraction Phenomenon: Why Multimodal MoEs See Without Thinking

Introduction

The recent paper "Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts" by Xu et al. identifies a subtle yet critical failure mode in large vision-language models that demands our attention. The authors document what they term the Seeing but Not Thinking phenomenon: multimodal Mixture-of-Experts (MoE) models can accurately perceive visual content, extracting every numerical value and textual element from an image, yet fail to solve problems they handle effortlessly when presented as pure text. Consider the arithmetic reasoning task illustrated in their work. When presented with a word problem about a beanstalk doubling in height, rendered as a clean high-resolution image, Qwen3-VL-30B-A3B correctly identifies all visual elements but produces a reasoning error. Presented with identical text, the model solves it perfectly. This discrepancy raises a fundamental question: why does visual perception impair reasoning even when the semantic content is fully extracted?

Isolating the Failure Mode

To systematically investigate this phenomenon, the authors constructed rigorous controlled experiments using the MATH500 dataset. They rendered pure-text mathematics problems as high-resolution images to ensure visual legibility, then compared model performance across modalities. The results are striking: between 68.2% and 73.1% of failures on visual inputs stem from reasoning errors rather than perception errors. Only 26.9% to 31.8% of cases involve actual misreading of the visual content. This quantifies the phenomenon precisely; the model sees correctly but thinks incorrectly.

A natural hypothesis attributes this to cross-modal semantic alignment failure. Perhaps visual information, though perceived, fails to map onto the textual semantic space in a way compatible with the model's reasoning circuits. The authors test this through cross-modal concept intervention experiments, manipulating hidden state representations across layers. They observe a characteristic inverted U-shaped pattern: intervention success rates are low in early layers where visual features remain unaligned, peak in middle layers where cross-modal semantic sharing occurs, and decline in terminal layers where output distributions are already determined. This confirms that MoE architectures do achieve cross-modal semantic sharing, ruling out representation misalignment as the primary culprit. The reasoning capability exists within the model; something prevents its activation during visual processing.

The Routing Mechanism as Culprit

Having eliminated semantic alignment, the investigation turns to the defining component of MoE architectures: the routing mechanism. Through systematic analysis of expert activation patterns across layers, the authors uncover a structural specialization that proves consequential. Visual experts exhibit high activation in early and terminal layers, presumably handling feature extraction and output formatting. Conversely, domain-specific reasoning experts cluster densely in middle layers. This architectural stratification creates a vulnerability.

The critical finding emerges when comparing routing patterns between text and image inputs. Image inputs induce significant routing divergence from text inputs precisely in these middle layers where domain experts concentrate. Greater divergence correlates strongly with lower reasoning accuracy across controlled conditions. This leads to the paper's central thesis, the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts, instead directing computation toward less suitable experts. The visual modality does not damage intrinsic reasoning capabilities; rather, it causes suboptimal expert selection that prevents utilization of domain-specific reasoning capacity.

Validation Through Intervention

To test this hypothesis, the authors designed a routing-guided intervention method that enhances domain expert activation during inference. The approach is elegant in its simplicity: if routing distraction causes reasoning failures, then explicitly increasing activation weights of domain-relevant experts should recover performance. Evaluating across three MoE models and six benchmarks, the soft routing guidance strategy yields consistent improvements. Even on tasks involving complex geometric figures and function graphs where visual information cannot be replaced by text, enhancing domain expert activation improves integration of perception with reasoning, with gains reaching 3.17% on complex visual reasoning tasks.

Further analysis reveals that domain expert identification is robust to information completeness. As long as reference text elicits target domain reasoning patterns, identified experts transfer effectively to visual tasks with different information structures. This suggests experts encode cognitive functions rather than memorizing sample-specific solutions. The model possesses general reasoning capabilities localized in specific experts; the routing mechanism simply fails to locate them when visual inputs dominate the attention landscape.

My Take: Architectural Implications and Future Directions

The routing distraction finding opens a clear next step that the paper suggests but does not fully explore: router-only fine-tuning. If routing weights alone account for the reasoning gap between text and image modalities, then targeted supervised fine-tuning of the router, using paired text and image examples of identical problems, should recover near-text performance at minimal computational cost. This would cleanly separate routing failure from representational failure. Current MoE training typically optimizes all parameters jointly, but this result suggests the router may require distinct training signals that explicitly penalize modality-induced routing divergence.

The 3.17% improvement from manual intervention, while modest, represents a lower bound. Automated methods for domain expert discovery, perhaps using gradient-based attribution or causal mediation analysis, could yield larger gains. More importantly, the layer-wise separation of visual and reasoning experts suggests a fundamental architectural tension. Current MoE designs may overly specialize experts by modality rather than by function, creating informational bottlenecks at the routing layer.

Several limitations warrant acknowledgment. The intervention method requires identifying domain experts beforehand, which may not scale to open-domain reasoning. Additionally, the experiments focus on mathematical reasoning; whether routing distraction similarly affects commonsense reasoning or visual question answering remains unclear. The correlation between routing divergence and accuracy, while strong, does not definitively establish causation without ablation studies that permanently modify routing weights rather than intervening during inference.

Conclusion

The work by Xu et al. reframes multimodal reasoning failures not as problems of perception or representation, but as coordination failures between routing mechanisms and expert selection. As MoE architectures become the dominant paradigm for scaling vision-language models, understanding these routing dynamics grows increasingly critical. The field must reconsider how routers are initialized and trained in multimodal contexts, potentially incorporating contrastive objectives that align routing distributions across modalities. The open question remains: can we design modality-agnostic routers that dispatch inputs to reasoning experts based on task requirements rather than input format? Answering this may require architectural innovations that decouple perceptual routing from cognitive routing, ensuring that seeing always enables thinking.