The Modality Gap in Vision-Language Models: When Adding Images Hurts Performance

One of the central assumptions driving multimodal artificial intelligence is that integrating additional sensory channels, particularly vision, should enhance reasoning capabilities. Yet a rigorous new study challenges this premise directly. In "Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap," Xu et al. demonstrate that state-of-the-art Vision-Language Models (VLMs) often achieve their highest reasoning performance when deprived of visual input entirely. More strikingly, providing both text and images frequently degrades performance compared to text alone. This phenomenon, termed the modality gap, suggests that contemporary VLMs are not genuine multimodal reasoners but rather text-centric systems that treat visual information as a noisy, secondary input stream.

Deconstructing the Modality Gap with CROSSMATH

To isolate the effect of modality on reasoning, the authors developed CROSSMATH, a benchmark designed around three core principles: tasks must be intrinsically vision-first, difficulty must be stratified, and information content must be strictly equivalent across modalities. The benchmark consists of 2D spatial grids containing intersecting mathematical equations, where models must infer missing numerical values based on geometric and algebraic constraints. Crucially, each problem is instantiated in three formats: a text-only markdown table, an image-only grid rendering, and a combined image-plus-text prompt. Human annotators verified that these formats contain identical task-relevant information, ensuring that performance differences reflect genuine reasoning disparities rather than informational asymmetry.

The results reveal a consistent hierarchy across tested models, including variants of Qwen and LLaVA. Text-only inputs consistently yield the highest accuracy, while image-only inputs produce the lowest. The counterintuitive middle case, image-plus-text, performs worse than text alone, indicating that visual integration actively interferes with reasoning rather than augmenting it. This degradation persists across difficulty levels, which the authors control by varying grid sizes (3x3 to 5x5), the number of missing equations, and operator complexity (addition through division). The consistency of this gap across model scales suggests the issue is architectural rather than merely a function of insufficient training data or model capacity.

The Architecture of Subservience

To understand why adding visual information impairs reasoning, one must examine the standard VLM pipeline architecture. Modern systems typically employ a vision encoder (usually a Vision Transformer) to extract visual features, an MLP projector to align these features with the latent space of a pre-trained Large Language Model (LLM), and finally the LLM itself for autoregressive generation. This design creates a fundamental asymmetry: visual information undergoes lossy compression into language-compatible tokens, while textual inputs enter the reasoning core natively.

The projector serves as a bottleneck. It translates high-dimensional visual features into a sequence of soft prompts that the language model can interpret, effectively reducing the rich spatial structure of an image into a constrained token budget. When both modalities are present, the model must attend to both native text tokens and these compressed visual representations. The observed performance degradation in image-plus-text conditions suggests that the attention mechanism may suffer from interference effects, or that the compression artifacts introduced by the vision encoder introduce noise that disrupts the LLM's otherwise pristine textual reasoning chains. In this framework, the vision encoder functions not as a reasoning engine but as a preprocessing layer that feeds a fundamentally text-centric cognitive architecture.

Your Take: Beyond the Training Data

The CROSSMATH findings force us to reconsider what we mean by "multimodal reasoning." The study's fine-tuning experiments provide particularly telling evidence. When the authors post-trained Qwen3.5-9B using Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) with exclusively image-based inputs, visual reasoning performance improved significantly. However, the modality gap persisted; text-only performance remained superior or equal even after vision-specific optimization. This suggests that the gap is not merely a curricular deficit that additional training can eliminate, but rather a structural feature of architectures that subordinate vision to language.

This has profound implications for fields requiring genuine visual-spatial reasoning, such as robotics, medical imaging, or scientific diagram interpretation. If current VLMs reason primarily in textual latent space, they may miss visual nuances that resist linguistic description, such as subtle textural gradients, precise spatial relationships, or dynamic physical interactions. The CROSSMATH tasks, while synthetic, embody the "vision-first" principle that many real-world domains require. The fact that models fail here indicates a limitation in how visual grounding is currently implemented.

One must also acknowledge the limitations of this study. CROSSMATH focuses on mathematical reasoning within controlled 2D grids, a domain where textual representations (tables) are naturally informationally dense. Natural photographs may exhibit different compression dynamics, where visual tokens capture information that would require prohibitively long text descriptions. Nevertheless, the benchmark's rigorous control of information equivalence sets a new standard for evaluation. Future work should extend this methodology to physics simulations, chemical diagrams, and natural scene understanding to test the generality of the modality gap.

Conclusion

The modality gap identified by Xu et al. serves as a critical diagnostic for the field. It reveals that current VLMs are, at their core, language models with visual input adapters rather than unified multimodal cognitive systems. The architectural decision to compress vision into the token space of a pre-trained LLM creates an inherent bottleneck that privileges textual reasoning.

Looking forward, the field faces a choice. We may continue to refine projection layers and training curricula to narrow the gap incrementally, accepting that vision will remain subservient to language. Alternatively, we might pursue native multimodal architectures that treat visual and textual representations as coequal throughout the reasoning process, potentially requiring training from scratch rather than grafting encoders onto frozen LLMs. The open question remains whether the next generation of models will reason about the world through the lens of language, or finally learn to see for themselves.