Steerable Visual Representations: Early Fusion for Vision-Centric Multimodality

Introduction

The modern computer vision landscape presents a curious dichotomy. On one hand, we possess powerful pretrained Vision Transformers such as DINOv2, MAE, and SigLIP that extract rich, generic visual features applicable across classification, segmentation, and retrieval tasks. On the other hand, these models suffer from a fundamental rigidity; they encode what is visually salient, typically the dominant object in a scene, with no mechanism to direct attention toward subtler elements. A model presented with a living room scene containing both a prominent cat and a diminutive remote control will invariably represent the cat, regardless of whether the downstream task requires locating the remote.

Conversely, Multimodal Large Language Models (MLLMs) offer remarkable flexibility through textual prompting. Yet this flexibility comes at a cost. By fusing vision and language in the later layers of a language model (late fusion), these architectures produce representations that are language-centric, often diminishing the visual fidelity necessary for precise spatial tasks. The research community has long faced a tradeoff between steerability and representation quality.

In their paper "Steerable Visual Representations," Ruthardt et al. propose a compelling third path. Their method, SteerViT, introduces a new class of visual representations that maintain the generality and spatial precision of pretrained ViTs while enabling text-based steering of both global and local features. The key innovation lies not in the concept of multimodal fusion itself, but in its timing and location within the processing pipeline.

The Architecture of Early Fusion

The technical distinction between SteerViT and preceding approaches centers on where and how language interfaces with vision. Traditional vision-language models like CLIP employ late fusion, encoding images and text independently before computing similarities in a joint embedding space. MLLMs such as Qwen3-VL or InternVL extend this paradigm, projecting visual tokens into the input space of a language model, thereby allowing cross-modal interaction only after the visual features have been fully extracted.

SteerViT inverts this logic by implementing early fusion within the visual encoder itself. The authors interleave lightweight cross-attention layers directly into the blocks of a frozen pretrained ViT, typically DINOv2. Rather than processing the image in isolation, the visual transformer blocks receive text embeddings through these cross-attention mechanisms at multiple depths. This architecture ensures that language influences the visual feature extraction process itself, not merely its interpretation.

The mechanism is both elegant and efficient. While the base ViT remains frozen, preserving its pretrained visual knowledge, only 21 million parameters (the cross-attention layers and associated projections) require training for adaptation. This represents a dramatic reduction compared to the 200 million to 1 billion trainable parameters typical of MLLMs or open-vocabulary localization models like SAM3. The authors introduce a gating factor, described in Equation 2 of the paper, which modulates the influence of text features at different layers. This gating allows for a tunable balance between steerability and representation quality, enabling the model to traverse a Pareto frontier that was previously inaccessible.

Crucially, this approach produces what the authors term "vision-centric multimodal representations." Unlike MLLMs, where visual information is subservient to the linguistic objective of next-token prediction, SteerViT maintains visual fidelity because the processing remains fundamentally within the visual encoder's architecture. The text acts as a conditioning signal, steering the visual features without reconstructing them into a language-compatible format.

Evaluation and the New Pareto Frontier

The authors establish rigorous benchmarks to measure what they define as the three desiderata of steerable representations: steerability (the ability to adapt to input text), representation quality (support for diverse visual tasks), and early vision-language fusion (architectural integration within the visual encoder).

For steerability, they employ text-guided image retrieval tasks, evaluating whether the model can retrieve images containing specific objects when prompted, even when those objects are small or visually subordinate. For representation quality, they measure linear probing accuracy on CLS tokens and semantic segmentation performance on patch features. The results reveal that SteerViT achieves a new Pareto frontier, simultaneously outperforming or matching dedicated approaches while maintaining the flexibility of text conditioning.

Particularly noteworthy are the zero-shot generalization capabilities demonstrated on anomaly detection and personalized object discrimination tasks. By steering the visual features toward specific object categories or individual instances, SteerViT matches specialized methods without task-specific fine-tuning. This suggests that the learned steering mechanism captures fundamental visual-semantic relationships that transfer across domains.

The comparison to Open-Vocabulary (OV) localization models like SAM3 is instructive. While SAM3 can be prompted to localize objects, it typically requires substantial trainable parameters and often sacrifices the generic utility of its visual features. SteerViT achieves comparable localization accuracy while preserving the underlying representation quality necessary for downstream tasks like segmentation, effectively decoupling the steering mechanism from the task-specific decoder.

Cognitive Parallels and Practical Implications

Beyond the technical specifications, SteerViT represents a conceptual alignment with human visual cognition. The authors note that human perception operates through top-down task-guided attention; priming a viewer with the word "bookshelf" fundamentally alters how they parse a visual scene, enhancing neural responses to bookshelf-related features even at early cortical stages. Late fusion architectures, by contrast, resemble the cognitive process of describing a photograph after viewing it, while SteerViT mimics the act of looking for something specific.

From a practical standpoint, this research suggests a future where vision models need not choose between generality and specificity. Current workflows often require either fine-tuning expensive models on narrow datasets or chaining together massive MLLMs for simple visual queries. SteerViT offers an intermediate solution: a lightweight, steerable visual backbone that maintains the efficiency of DINOv2 while accepting natural language guidance. For applications in robotics, autonomous navigation, or interactive image editing, where systems must rapidly shift attention between objects based on user intent or task requirements, such steerability eliminates the need for model retraining or cumbersome prompt engineering within massive language models.

However, limitations remain apparent. The method's effectiveness depends on the quality of the frozen pretrained ViT; SteerViT cannot extract features that the base model never learned to represent. Additionally, while the gating mechanism provides control over text influence, determining optimal gating values for novel tasks may require empirical calibration. The paper also focuses primarily on single-object steering; complex scenes requiring simultaneous attention to multiple, disparate objects present an ongoing challenge for the cross-attention mechanism.

Conclusion

Steerable Visual Representations mark a significant evolution in how we conceptualize the relationship between vision and language in artificial systems. By demonstrating that early fusion within the visual encoder can preserve representation quality while enabling flexible text conditioning, Ruthardt et al. challenge the prevailing assumption that effective multimodal interaction requires massive language models or late-stage fusion.

The work raises intriguing questions for future research. Can this steering mechanism extend to temporal domains, enabling text-guided video understanding without frame-by-frame processing? How might steerable representations interact with generative models, potentially allowing users to steer not just perception but synthesis toward specific visual concepts? Furthermore, the success of parameter-efficient adaptation (21 million trainable parameters) suggests broader implications for the field of model efficiency and continual learning.

As vision models become foundational infrastructure across industries, the ability to steer their attention without retraining or sacrificing visual fidelity transitions from a convenience to a necessity. SteerViT provides a technical blueprint for this capability, offering a path toward computer vision systems that combine the reliability of pretrained feature extractors with the flexibility of natural language interaction.