Directional Confusions and the Geometry of Inductive Bias in Human and Machine Vision

Introduction

Accuracy has long served as the primary currency for comparing human and machine perception. When a deep convolutional network achieves ImageNet accuracy rivaling human performance on natural images, we are tempted to conclude that the underlying representations have converged. Yet the paper Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision by Caglar, Mediano, and Lin challenges this assumption through a subtle but crucial observation. The authors demonstrate that humans and deep vision models can exhibit identical accuracy levels while displaying systematically different error structures. Specifically, they differ in directional confusions: who gets mistaken for whom, and in which direction.

This distinction matters because directional asymmetries reveal inductive biases invisible to aggregate metrics. When a model confuses a zebra for a barcode but not vice versa, or collapses multiple dog breeds into a single Labrador prototype, it reveals something fundamental about how the system carves up perceptual space. Drawing on classic cognitive science work regarding asymmetric similarity judgments (Tversky and Gati, 1982; Rosch, 1975), the authors argue that these patterns are not noise but diagnostic signatures of representational geometry. By leveraging a rate-distortion framework grounded in information theory, they show that error directionality serves as a compact, interpretable signature of how systems generalize under distribution shift.

Quantifying Directional Bias Through Rate-Distortion Geometry

The study employs a natural-image categorization task subjected to twelve distinct perturbation types, including texture corruptions, adversarial noise, and naturalistic distortions. The researchers collect matched behavioral responses from human participants and deep vision models, preserving the directional structure of confusion matrices rather than symmetrizing them. This preservation is crucial because it captures phenomena such as prototype effects and category typicality, where similarity relations violate metric axioms. For instance, just as humans judge a robin to be more similar to a bird than the converse, vision models may exhibit one-way diagnostic dependencies that expose their reliance on specific feature shortcuts.

To formalize these patterns, the authors adopt a rate-distortion (RD) framework treating each vision system as an effective communication channel. The confusion matrix defines the channel characteristics, mapping stimuli to responses with specific conditional probabilities. The RD curve then characterizes the fundamental trade-off between the mutual information rate (the complexity of the internal representation) and the expected categorical distortion (the error rate). This