Entropy as a Compass: Rethinking Visual Grounding in Vision-Language Models

Vision-language models have achieved impressive performance on standard benchmarks, yet they consistently fail when confronted with fine-grained visual reasoning or compositional queries requiring evidence from spatially disjoint regions. Whether locating small text in high-resolution documents, identifying symbols in cluttered scenes, or aggregating clues spread across multiple image areas, these systems often miss the very details that determine correct answers. In "Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models," Gröpl and colleagues from ETH Zürich and KAIST propose a fundamental shift in how we approach visual localization. Rather than treating grounding as a supervised detection task or mining attention maps for heuristic saliency, they frame inference as test-time evidence acquisition guided by the model's own uncertainty. This approach addresses what the authors term "selective perception," the capacity to actively identify decisive visual evidence rather than merely generating plausible text conditioned on holistic image understanding.

From Attention Maps to Entropy Gradients

Current training-free grounding approaches typically rely on attention map heuristics, which suffer from significant limitations. Methods like ViCrop extract localization cues from attention patterns across forward passes, then commit to a single crop based on heuristic saliency thresholds. This creates a fundamental tension: attention maps tend to collapse multi-region evidence into a single dominant blob, and selecting informative heads requires arbitrary choices that vary unpredictably across architectures. When queries require aggregating spatially disjoint evidence, such as relating a figure caption to its distant reference in a document, attention-based methods frequently fail to capture secondary regions.

The entropy-gradient method operates on a different principle entirely. At a single decoding step, the authors compute the Shannon entropy of the next-token distribution, H = -Σ p(x) log p(x), which quantifies the model's uncertainty about its upcoming prediction. They then backpropagate this scalar quantity to the visual token embeddings, obtaining ∇_v H, the gradient with respect to visual features. The resulting entropy-gradient map indicates precisely which visual regions most influence the model's uncertainty. Regions with high gradient magnitude represent information bottlenecks; modifying these features would most significantly alter the prediction distribution. This produces a grounding signal tightly coupled to the model's decision-making behavior, without requiring auxiliary detectors or attention-head selection heuristics.

Crucially, this uncertainty-driven signal naturally supports multi-region localization. The authors convert the entropy-gradient response into multiple coherent regions of interest, enabling explicit aggregation over spatially disjoint evidence. This capability proves essential for document understanding and compositional visual question answering, where answers depend on relationships between separated visual elements rather than single salient objects.

Iterative Refinement and Empirical Validation

While entropy gradients provide superior localization signals, single-pass maps may still suppress fine-grained details beneath dominant activations. To address this, Gröpl et al. introduce an iterative zoom-and-reground procedure that reapplies the entropy-backpropagation mechanism on selected subregions. Upon identifying a coarse region with high entropy gradient, the system extracts that crop and recomputes the gradient within the zoomed