Action Blindness: Why Embodied Spatial Intelligence Requires More Than Better Perception

When we talk about spatial intelligence in machine learning, we usually frame it as a perception problem. Give the model better depth estimation, sharper segmentation, more accurate 3D reconstruction, and the spatial reasoning will follow. The recent paper ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop argues that this framing misses the point entirely. The interesting bottleneck is not what the agent can see but what the agent chooses to look at and how it moves to reveal what is hidden.

This is a meaningful reframing, and it deserves careful attention. Below I unpack the benchmark, the experimental findings, and what I think they imply for the broader trajectory of embodied AI research.

From Spatial Sensing to Spatial Competence

The central conceptual move in ESI-Bench is to recast the observer as an actor. Most prior spatial reasoning benchmarks assume what the authors call oracle observations: the agent is handed a fixed image, or a fixed set of images, and asked to reason about the scene. This implicitly treats spatial intelligence as a function from pixels to answers.

But cognitive science has long characterized perception as perceptually guided action, a notion the paper traces back to Gibson and to the sensorimotor account of O'Regan and Noë. The core idea is that knowing how observations change as a function of action is often harder than the sensing itself. A question like "is the chestnut inside the glass?" or "how many apples are in this scene?" under heavy occlusion cannot be resolved by staring harder at one viewpoint. The agent has to move, probe, or manipulate.

ESI-Bench operationalizes this with 10 task categories, 29 subcategories, and 3,081 task instances, all built on the OmniGibson simulator. The taxonomy is organized around Spelke's four core knowledge systems from developmental psychology:

• Object representation (containment, transparency, occlusion)
• Layout and geometry (spatial relations, geometric configuration, navigation)
• Number representation (counting under occlusion, enumeration)
• Agents and goal-directed actions (action sequencing, stacking and stability)

What makes the design clever is that these questions are constructed to be unanswerable from passive observation alone. Counting apples behind an occluder, judging whether an object is real or a reflection in a mirror, inferring an unobserved change inside a closed box. Each requires the agent to decide which embodied ability to deploy, perception, locomotion, or manipulation, and in what order.

The Empirical Findings That Matter

The authors evaluate state of the art multimodal large language models across three paradigms: passive single-view, passive random multi-view, and active exploration, with a ground truth oracle to disentangle perception errors from action errors. Several results stand out.

Active exploration substantially outperforms passive sensing. This is unsurprising in direction but notable in magnitude. More importantly, the agents discover emergent spatial strategies without explicit instruction. They spontaneously compose actions, moving to a better vantage point, then probing, in ways the prompt never specified. This suggests that the latent capability to plan informative action sequences exists inside these models and can be elicited when the interaction loop is opened up.

Random multi-view often adds noise rather than signal. This is the finding I find most instructive. Passive random multi-view consumes far more images yet frequently degrades performance relative to a single well-chosen view. The lesson generalizes well beyond this benchmark: more observation is not better observation. Feeding a model a dense pile of redundant or uninformative frames dilutes the task-relevant evidence and can actively confuse the reasoning process. The value of an observation is not in its existence but in its information gain conditioned on the question.

Most failures are action blindness, not perception failure. Using the oracle baseline to separate the two error sources, the authors show that the dominant failure mode is not weak perception. The agent often sees fine. It simply makes poor action choices, which produce poor observations, which then drive cascading downstream errors. The perception module is not the weak link; the policy that decides what to perceive is.

Imperfect 3D grounding can be worse than 2D. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. This is an important caution against the assumption that lifting everything into a 3D representation is automatically beneficial. A noisy 3D estimate introduces systematic geometric errors that a model treats as ground truth, whereas a 2D baseline at least keeps the model appropriately uncertain.

The Metacognitive Gap

The most thought-provoking part of the paper is the human study. Humans, when faced with spatial ambiguity, seek falsifying viewpoints. They actively look for the observation that would prove their current hypothesis wrong, and they revise their beliefs when contradicted. This is Popperian behavior baked into ordinary perception.

The models do not do this. They commit prematurely with high confidence regardless of evidence quality. They do not treat an observation as a test of a hypothesis; they treat the first plausible reading as the answer. The authors call this a metacognitive gap, and they argue, correctly in my view, that neither better perception nor more embodied interaction alone can close it.

This connects to a recurring theme in the literature on calibration and overconfidence in large models. The failure here is not that the model cannot acquire the disambiguating evidence. It is that the model does not know that it should, because it does not maintain a belief state with an associated uncertainty that would motivate further information seeking. Active exploration only helps if the agent has a reason to keep exploring, and that reason is uncertainty it actually represents and acts on.

My Take

I think ESI-Bench is pointing at something that the field has been circling for a while: the separation between a perception model and a policy over perception is artificial, and our benchmarks have been quietly optimizing only the former. The reframing from spatial sensing to spatial competence is the right one, and the oracle ablation is the methodological contribution that makes the argument credible rather than merely rhetorical. Without that ablation, you could always blame perception.

Here is an inference the paper does not state directly but that I think follows. The action blindness result implies that the bottleneck is closer to active inference than to representation learning. The agent needs an internal notion of which actions reduce uncertainty about the answer, which is essentially a value of information computation. Current MLLMs have no native mechanism for this. They were trained on static text and image data where no action ever changed the available evidence. It would be surprising if next-token prediction over passive corpora produced strong information-seeking policies, and the empirical results confirm it did not.

This also reframes the random multi-view result. The reason dense sampling hurts is that it substitutes blind coverage for targeted inquiry, and a model with no information-value sense cannot tell the difference between a useful frame and a redundant one. Coverage is a poor proxy for information when you cannot weight what you collect.

There are limitations worth noting. The benchmark lives in OmniGibson, so the sim-to-real gap is unaddressed, and real-world perception noise could shift the balance between action errors and perception errors. The taxonomy, while principled through Spelke's systems, is still a human-curated selection of "hard" tasks, which introduces a selection effect toward exactly the questions that punish passive sensing. And the metacognitive gap, while clearly demonstrated, is not yet mechanistically explained.

Open Questions

The forward-looking question is how to instill the value of information into these systems. Do we need explicit uncertainty representations and information-gain objectives, or can the right reinforcement learning signal over embodied rollouts induce falsification-seeking behavior implicitly? Can we transfer the metacognitive humility that humans display, or is it an emergent property of training under physical stakes that no amount of passive pretraining will supply?

ESI-Bench does not answer these, but it gives the field a clean instrument to measure progress on them. That, more than any single number, is its contribution. Intelligence here is not about processing what is seen. It is about deciding what to seek next, and knowing when you do not yet know enough.