The Binding Problem in Textual Space: Why LLMs Fail at Viewpoint Rotation Despite Encoding Correct Spatial Representations

Introduction

Spatial intelligence has long been considered a cornerstone of artificial general intelligence. While recent advances in vision language models have demonstrated impressive capabilities in visual spatial reasoning, a fundamental question remains largely unexplored. Can large language models possess spatial intelligence when deprived of visual input, relying solely on linguistic descriptions of spatial relationships?

A recent study by Yang et al., titled "How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study", investigates this precise question through the lens of viewpoint rotation understanding (VRU). The researchers constructed VRUBench, a dataset where models must track their virtual orientation through a sequence of textual descriptions involving rotations (e.g., "turn right 90 degrees") and observations (e.g., "you see a book"), ultimately predicting the object visible from the final viewpoint.

The results reveal a striking capability gap. Human participants achieve perfect accuracy on this task, yet state of the art models including Qwen3-VL struggle to reach 60 percent accuracy. This discrepancy becomes even more puzzling when considering that VLMs consistently outperform their purely textual counterparts (LLMs) even when no visual input is provided, suggesting that visual pretraining somehow enhances abstract spatial reasoning.

The Encoding Paradox in Middle Layers

To understand why models fail despite having access to the complete textual description, Yang et al. conducted comprehensive mechanistic interpretability analyses. Using layer wise probing, they discovered that models actually possess strong capabilities for encoding spatial information in their hidden states. Early to middle layers successfully represent both the relative rotations (direction and angle) and the absolute orientation of the viewpoint.

This finding presents a paradox. If the model accurately encodes where it is oriented in space, why does it hallucinate incorrect observations in the output? The answer lies in the transition from representation to decision making. The probing accuracy for absolute orientation peaks in middle layers but diminishes in later layers, suggesting a handoff from spatial perception to response generation.

To trace this failure precisely, the authors employed path patching, a causal intervention technique that allows researchers to identify which attention heads are