When Text Surpasses Vision: The Counterintuitive Superiority of LLMs over VLMs in 3D Spatial Evaluation

Introduction

The prevailing wisdom in artificial intelligence holds that spatial reasoning is the natural domain of vision. When we evaluate whether a generated 3D indoor scene is plausible, the instinctive approach is to render the scene and ask a Vision-Language Model (VLM) to look at it. Vision, after all, is how humans verify spatial arrangements. Yet this assumption may be fundamentally misaligned with the nature of structured spatial reasoning. In "SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis," Sengupta, Ao, and Cascante-Bonilla demonstrate that text-only Large Language Models (LLMs) can outperform VLMs in assessing semantic layout quality, while VLM-based evaluators exhibit alarming instability when judging the same scene from different viewpoints.

This finding challenges the hierarchy of modalities we often take for granted. The work introduces SceneCritic, a symbolic evaluator that operates directly on floor-plan layouts rather than rendered observations, leveraging SceneOnto, a structured spatial ontology constructed from 3D-FRONT, ScanNet, and Visual Genome. By traversing explicit relational constraints encoding object co-occurrence probabilities, scale distributions, and canonical orientations, SceneCritic provides object-level diagnostic assessments that prove more stable and human-aligned than neural judges. The implications extend beyond scene generation; they force us to reconsider how we match evaluation modalities to the underlying structure of the tasks we seek to measure.

The Instability of Vision-Based Evaluation

Current standards for evaluating 3D indoor scenes rely heavily on VLMs judging a small number of rendered views. This methodology, while convenient, introduces significant methodological vulnerabilities that undermine its validity as a metric. The paper illustrates this with a striking example: when evaluating a piano room containing nine objects, a VLM assigns scores of 75% and 55% to the same scene depending solely on which view is rendered. This 20-point variance is not anomalous; it reflects systematic sensitivities to viewpoint selection, occlusion patterns, and prompt phrasing that make VLM evaluators unreliable arbiters of spatial quality.

The instability arises from a fundamental mismatch between the representation being evaluated and the perceptual mechanism doing the evaluation. Floor-plan layouts encode explicit geometric and semantic relationships (object A is 2.5 meters from object B, oriented toward it, and scaled appropriately), while rendered views introduce incidental visual properties (lighting artifacts, texture details, perspective distortion) that are irrelevant to spatial plausibility. When a VLM evaluates a top-down view versus a frontal view, it may miss occluded relationships or hallucinate constraints that do not exist in the underlying layout. The paper notes that such evaluators are "prone to hallucination" and highly sensitive to minor prompt rephrasing, rendering them unsuitable for tracking incremental improvements in generation models. If the evaluator's score depends more on camera angle than on the actual spatial configuration, we cannot determine whether our models are learning to construct coherent scenes or simply learning to exploit the evaluator's perceptual biases.

SceneOnto and the Mechanics of Symbolic Evaluation

SceneCritic addresses these limitations by bypassing rendering entirely and evaluating the intermediate layout representation directly. At its core lies SceneOnto, a spatial ontology built by aggregating indoor scene priors across multiple datasets. Unlike learned embeddings or procedural rules designed for generation, SceneOnto constructs explicit relational graphs of conditional co-occurrence probabilities, scale distributions per furniture category, and canonical orientations relative to room types and other objects. This structure enables SceneCritic to verify semantic, orientation, and geometric coherence across object relationships through symbolic traversal rather than perceptual inference.

The evaluation framework operates along three distinct axes. Semantic coherence checks whether object co-occurrences and scales reflect plausible indoor configurations (for instance, verifying that a table's height falls within the expected 0.308 to 1.191 meter range based on aggregated dataset statistics). Orientation correctness verifies that objects face appropriate referents (chairs facing tables they co-occur with, for example). Geometric overlap detection identifies physical collisions that would violate physical plausibility. Crucially, SceneCritic provides object-level and relationship-level assessments rather than global scores. It can identify that Chair 0 violates orientation constraints by not facing its associated table, while simultaneously confirming that Chair 1 is correctly placed. This granularity allows practitioners to diagnose specific failure modes rather than receiving opaque numerical ratings.

The paper demonstrates that these symbolic constraints align substantially better with human judgments than VLM-based evaluators. This alignment suggests that human spatial reasoning relies on symbolic and relational knowledge rather than purely perceptual matching. When we judge whether a room layout makes sense, we appear to verify conceptual constraints (can a piano fit in this corner? should these chairs face each other?) rather than assessing visual realism.

The Modality Paradox: Text vs. Vision in Spatial Assessment

Perhaps the most counterintuitive finding in this work concerns the comparison between LLM and VLM critics in the iterative refinement testbed. The authors probe how models build and revise spatial structure under three critic modalities: a rule-based critic using collision constraints, an LLM critic operating on the layout as text, and a VLM critic operating on rendered observations. While intuition suggests that vision would provide richer feedback for spatial tasks, the results indicate that text-only LLMs can outperform VLMs on semantic layout quality.

This inversion occurs because VLMs are distracted by the very perceptual richness that makes them powerful for other tasks. Rendering artifacts, viewpoint bias, and the loss of metric information during projection from 3D to 2D introduce noise that obscures the underlying spatial structure. An LLM operating on a structured text representation of object positions, dimensions, and categories reasons directly over the symbolic relationships that constitute spatial coherence. Without the perceptual noise of rendering, the LLM can focus on the semantic and geometric constraints that actually define a plausible scene.

However, the picture is nuanced. The paper notes that image-based VLM refinement proves most effective for semantic and orientation correction during iterative refinement. This suggests that while VLMs are unreliable as static evaluators (due to the viewpoint sensitivity discussed earlier), they can provide valuable gradient information when used interactively to guide refinement. The distinction appears to be between evaluation (judging a final state) and guidance (steering toward improvement). VLMs may excel at the latter while being poorly suited for the former.

My Analysis: Representation and the Illusion of Visual Understanding

This research exposes a deeper methodological trap in modern AI evaluation: the assumption that more powerful perceptual modalities automatically yield better judgment for structured tasks. We have conflated the ability to see with the ability to understand spatial relationships. A VLM can describe what a room looks like; it cannot necessarily verify whether the spatial graph of object relationships satisfies the constraints of physical and semantic plausibility. The perceptual richness that enables VLMs to generate detailed captions becomes a liability when we need precise, view-invariant verification of geometric and semantic constraints.

The superiority of symbolic and text-based evaluation for layout quality suggests that spatial reasoning is fundamentally a symbolic operation. When we evaluate whether a chair faces a table appropriately, we are checking a relational predicate against a canonical orientation distribution, not performing visual pattern matching. SceneCritic's approach reveals that much of what we consider visual understanding in spatial tasks is actually structured relational reasoning that can be better encoded in explicit ontologies than learned from pixels.

This has significant implications for embodied AI and robotics. As we train agents to navigate and manipulate objects in realistic spaces, we require evaluation metrics that diagnose specific failures in spatial reasoning rather than providing noisy aggregate scores. SceneCritic's object-level violation reporting provides exactly this diagnostic capability. However, we must also acknowledge limitations. SceneCritic's constraints derive from dataset statistics and may penalize novel but valid spatial configurations that fall outside typical distributions in 3D-FRONT or ScanNet. The ontology approach risks encoding conservative priors that could hinder exploration of creative or unconventional layouts.

Conclusion

SceneCritic represents a necessary corrective to the field's overreliance on neural judges for structured spatial tasks. By demonstrating that text-only LLMs can surpass VLMs in evaluating semantic layout quality, and by providing a stable, interpretable alternative to view-dependent evaluation, this work challenges us to align our evaluation modalities with the underlying structure of spatial reasoning. The instability of VLM evaluators is not merely a technical inconvenience; it is a symptom of a deeper representational mismatch.

Looking forward, the field must develop hybrid approaches that combine the symbolic stability of rule-based evaluators with the flexibility of learned models. Can we construct neuro-symbolic critics that use neural networks to extend ontological coverage while maintaining the view-invariance and diagnostic precision of SceneCritic? Can we develop VLM-based evaluation protocols that explicitly account for viewpoint bias through multi-view consistency checks? The paper's iterative refinement testbed suggests that the interaction between symbolic constraints and neural guidance holds particular promise. As 3D scene generation becomes central to embodied AI, robotics, and virtual environments, the development of trustworthy, interpretable evaluation frameworks is not merely a methodological nicety. It is a prerequisite for scientific progress.