When Video Models Answer Correctly but Understand Nothing: A New Benchmark for Physical Grounding

The Specter of Ungrounded Competence

Large vision language models can now describe complex physical events with apparent sophistication. Ask a model about a video showing liquid pouring into a container, and it will likely identify the action correctly. But does it actually see the liquid moving through space and time, or has it merely learned that the word "pouring" statistically correlates with certain visual textures and container shapes?

This distinction between correct answers and grounded understanding sits at the heart of "Grounding Video Reasoning in Physical Signals," a recent study by Osmanli, Cheng, and Gong from Queen Mary University of London. The work exposes a troubling pattern in current video question answering benchmarks. When a model achieves high accuracy on standard datasets, we assume it understands the video content. The reality is more concerning. Models can achieve correct answers by exploiting textual regularities in prompts while failing entirely to localize events temporally or spatially. A question about a collision might be answered correctly because the word "collision" appears in the prompt, not because the model identified where and when the impact actually occurred.

This matters profoundly for physical reasoning. Unlike abstract visual recognition, physical understanding requires tracking motion, contact, force, and state change across time and space. In robotics and embodied AI, a model that answers correctly but grounds poorly represents a liability rather than an asset. The study extends the diagnostic framework of V-STaR, which previously separated "what," "when," and "where" understanding, into six physics domains: Gravity, Fluids, Collisions, Deformation, Friction, and State Changes.

The Benchmark Architecture: Controlling for Linguistic Mirroring

The technical contribution of this work lies in its meticulous control structure. The researchers constructed a dataset of 1,560 base video clips drawn from four diverse sources: Something Something V2, YouCook2, HoloAssist, and Roundabout-TAU. Rather than treating question generation as a secondary annotation task, they established a shared grounded event record for each clip. This record specifies the event description, the temporal span of the physical interaction, and the spatial bounding boxes of participating objects.

From this single event record, the authors derive three distinct prompt families. The physics family uses explicit physical terminology and represents the main evaluation regime. The vstar_like family adapts V-STaR style semantic queries, serving as a non-physics comparison that still requires visual understanding. The neutral_rstr family functions as a templated control with neutral wording, essentially testing whether models can succeed when physical cues are minimized in the language.

Crucially, while the semantic target for "what" varies appropriately across these families, the temporal ("when") and spatial ("where") targets remain fixed. This design isolates the contribution of linguistic framing from visual grounding. If a model performs well on physics prompts but fails on neutral_rstr despite identical visual content and identical ground truth localization targets, we can conclude its success depends on linguistic shortcuts rather than visual analysis.

The evaluation extends further into robustness through four input conditions: original videos, temporally shuffled frames, ablated frames, and frame-masked inputs. This perturbation framework tests whether models rely on temporal continuity or specific visual evidence, or whether they can maintain grounding when sensory input degrades.

Selective Robustness and the Spatial Bottleneck

The empirical results reveal a fragmented landscape of competence that aggregate accuracy metrics obscure entirely. Across models and prompt families, physics prompts consistently yield the strongest performance, followed by vstar_like, with neutral_rstr serving as the hardest control. This ordering suggests that explicit physical language helps models align their internal representations with visual content, or alternatively, that neutral wording removes helpful priors that models have learned to exploit.

More revealing is the finding that prompt-family robustness is selective rather than universal. Two models with similar aggregate scores can exhibit diametrically opposed behavior when prompt framing shifts. Some models maintain relatively stable performance across all three families, suggesting genuine visual grounding. Others collapse completely when the linguistic scaffolding of physics terminology is removed. This selectivity implies that "video understanding" is not a monolithic capability but a patchwork of strategies, some visual and some textual.

The perturbation analysis yields perhaps the most counterintuitive result. Performance gains under degraded input conditions, such as shuffled or masked frames, cluster disproportionately in models with weaker original-condition baselines. Strong models show smaller perturbation gains, suggesting they already extract maximal information from intact video streams. Weaker models benefit from perturbations not because they are robust, but because the perturbations disrupt misleading temporal correlations or force the model to attend more carefully to remaining evidence. These gains function as diagnostics of evidence sensitivity rather than simple robustness metrics.

Across all settings, spatial grounding remains the most persistent weakness. Models struggle more with "where" questions than "when" questions, and both lag behind "what" recognition. This spatial bottleneck aligns with known limitations in current visual encoders, which often prioritize semantic classification over precise localization. In physical domains, however, spatial precision is not optional. Understanding a collision requires knowing which objects contacted, not merely that contact occurred somewhere in the frame.

Original Insights: The Semantics of Physical Causation

Beyond the paper's explicit findings, the benchmark design invites deeper reflection on the relationship between language and physical reasoning in multimodal systems. The three prompt families effectively create a spectrum of causal explicitness. Physics prompts make causal mechanisms transparent; they name forces, materials, and interactions directly. Neutral prompts obscure these mechanisms, forcing the model to infer physical causation from raw visual dynamics.

The selective robustness observed across models suggests that current video LLMs occupy different positions on a spectrum from "language dominant" to "vision dominant" reasoning. Language dominant models treat video as confirmation of hypotheses generated from prompt text. Vision dominant models treat text as a query against visually extracted physical representations. The latter are preferable for embodied applications, yet standard benchmarks cannot distinguish between them.

The spatial grounding weakness carries specific implications for robotics. A manipulation system that knows "pouring" is occurring but cannot localize the stream of liquid relative to the cup rim cannot execute the action or verify its success. The benchmark's insistence on bounding box evaluation for physical events thus addresses a critical gap between video understanding and embodied intelligence.

Furthermore, the perturbation framework suggests a path toward more rigorous evaluation of video models for deployment in variable conditions. Real world robotics operates with occluded views, dropped frames, and asynchronous sensory input. A model that gains performance when frames are shuffled or masked is likely exploiting dataset artifacts or temporal biases rather than building coherent physical representations. The paper's diagnostic indices for perturbation analysis should become standard practice in video LLM evaluation.

Conclusion: Toward Mechanistic Understanding

The work of Osmanli, Cheng, and Gong provides more than a new benchmark. It establishes a methodological standard for determining whether video models understand physical events or merely name them. The evidence suggests that current models, despite impressive aggregate scores, remain vulnerable to linguistic shortcutting and spatial imprecision.

Looking forward, the field must move beyond accuracy metrics toward mechanistic understanding. We need to know not just whether models answer correctly, but whether their internal attention maps correlate with physical objects, whether their temporal reasoning tracks causal chains rather than mere succession, and whether their spatial localization is precise enough for physical interaction.

The six physics domains covered here, gravity, fluids, collisions, deformation, friction, and state changes, represent fundamental building blocks of intuitive physics. Mastery of these domains with genuine spatial and temporal grounding may serve as a necessary precondition for trustworthy embodied AI. This benchmark offers the diagnostic tools to measure that mastery honestly, even when the results reveal uncomfortable gaps between apparent competence and genuine understanding.