Unbiasing Vision Transformers: How ALiBi Positional Encoding Rescues Scientific Imaging from DINOv2's Spatial Assumptions

Vision transformers have become the default architecture for visual representation learning, with models like DINOv2 providing remarkably general features for downstream tasks. Yet beneath their impressive performance on ImageNet and COCO lies a subtle architectural assumption that silently corrupts their utility in scientific domains. In their recent study, "What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers," Pawlowsky et al. expose how the learned positional encodings in self-supervised ViTs encode absolute spatial coordinates as artefacts unrelated to image content. For natural photographs, this bias remains largely invisible; for microscopy, satellite imagery, or medical scans, where no spatial direction carries inherent semantic privilege, it proves actively harmful. The authors propose a surgical replacement of learned positional encodings with 2D-aware ALiBi relative biases, demonstrating that this intervention preserves semantic richness while eliminating the pernicious spatial assumptions that plague zero-shot segmentation in homogeneous materials.

The Anatomy of Positional Bias in Self-Supervised ViTs

The standard vision transformer architecture processes images by flattening non-overlapping patches into a sequence, then adding positional encodings to inject spatial awareness into the attention mechanism. Most foundation models, including DINOv2 and MAE, employ learned positional embeddings; vectors trained end-to-end that ostensibly teach the model where each patch resides in the image grid. Pawlowsky et al. reveal through linear probing that these encodings do not merely provide geometric context. Instead, they create dedicated channels in the final feature representations that function as pure positional ramp functions, encoding absolute coordinates regardless of input content.

This phenomenon proves surprisingly specific to self-supervised models. When the authors compared DINOv2 and MAE against supervised ViTs trained on labeled ImageNet data, they found that the supervised models exhibited significantly less positional bias in their latent representations. The implication is stark: self-supervised objectives, which rely on reconstruction or contrastive learning between augmented views, somehow incentivize the model to memorize absolute spatial locations as auxiliary tasks. In DINOv2, certain feature channels effectively broadcast "this is the top left corner" or "this is the center" irrespective of whether those regions contain sky, tissue, or metal grain boundaries.

For natural images, such biases might even prove beneficial. Photographs exhibit strong spatial priors; gravity pulls objects downward, horizons align horizontally, and faces center in portraits. However, materials science imaging fundamentally violates these assumptions. Scanning electron microscopy (SEM) and transmission electron microscopy (TEM) produce grayscale cross-sections of homogeneous microstructure where a grain boundary in the upper left corner carries identical semantic weight to one in the lower right. When DINOv