The Enumeration Gap: Why Counting in Long Videos Remains Unsolved

Introduction

Counting is among the earliest mathematical skills humans acquire, yet it remains an unsolved frontier for multimodal AI. While a child can reliably tally occurrences of an event across a thirty minute video, state of the art multimodal large language models (MLLMs) fail at this task with surprising consistency. This disparity is exposed by EC-Bench: Enumeration and Counting Benchmark for Ultra-Long Videos, a rigorous evaluation framework that reveals fundamental limitations in how current systems process extended temporal contexts. The benchmark demonstrates that accurate quantitative reasoning in long form videos requires more than pattern matching; it demands systematic enumeration, precise temporal grounding, and the maintenance of object permanence across sparse, heterogeneous events.

The Architecture of Failure

EC-Bench distinguishes itself through a tripartite evaluation structure that treats counting not as a single numerical output, but as a pipeline involving enumeration, temporal evidence grounding, and final aggregation. The dataset comprises 152 untrimmed videos exceeding thirty minutes each, annotated with 1,699 queries across four distinct reasoning categories: Conditional Counting (instances satisfying temporal or spatial constraints), Parallel Event Counting (aggregating occurrences across disjoint time intervals), Appearance Counting (listing distinct visual instances such as brands or objects), and Spatial Counting (enumerating items within specific visual regions).

This design exposes a critical methodological flaw in previous benchmarks. Prior evaluations typically provided short, trimmed clips and accepted only numerical answers, obscuring whether models correctly identified relevant instances or merely guessed plausible quantities. EC-Bench requires models to explicitly enumerate instances and localize them within precise temporal spans, creating an interpretable audit trail from perception to conclusion.

The results are sobering. Across 22 evaluated MLLMs, the highest performing system achieved merely 29.98% accuracy on enumeration tasks and 23.74% on counting tasks. Human annotators, by contrast, scored 78.57% and 82.97% respectively. This performance chasm suggests that current architectures process long videos as extended sequences of independent frames rather than maintaining coherent object trajectories across time.

The Temporal Grounding Bottleneck

The paper's most significant finding concerns the relationship between enumeration accuracy, temporal grounding precision, and counting performance. Analysis reveals that these three capabilities are strongly correlated; models that fail to correctly identify when events occur invariably fail to count them accurately. This indicates that the primary bottleneck in long video counting is not numerical aggregation, but rather the maintenance of temporal coherence and instance discrimination across extended durations.

This finding challenges the prevailing assumption that larger context windows alone will solve long video understanding. While recent MLLMs support very long input sequences, EC-Bench shows that simply increasing the number of sampled frames yields limited improvements in counting accuracy. The issue is not the quantity of visual information processed, but the quality of temporal reasoning applied to that information. Models appear to suffer from a form of temporal attention decay, where events occurring early in a video are either forgotten or conflated with similar events occurring later, leading to duplicate counts or omissions.

The benchmark's requirement for explicit evidence spans further illuminates the nature of model errors. When asked to provide temporal intervals supporting their counts, MLLMs frequently hallucinate evidence spans or conflate temporally distant occurrences of similar visual patterns. This suggests that current systems lack robust mechanisms for temporal indexing and instance reidentification, capabilities essential for any genuine understanding of extended video narratives.

Beyond Pattern Matching

The enumeration gap identified by EC-Bench has profound implications for the field's theoretical understanding of video comprehension. Enumeration serves as a minimal viable test for object permanence and temporal consistency; to correctly list all instances of an object across thirty minutes of video, a system must maintain stable representations of object identity despite occlusion, viewpoint changes, and temporal gaps. The failure of current MLLMs on this task indicates that these systems are performing sophisticated pattern matching without constructing the structured, temporally persistent world models that underlie human visual reasoning.

This limitation is particularly concerning for applications requiring quantitative verification, such as sports analytics, surveillance review, or scientific observation. In these domains, numerical accuracy is not merely a benchmark metric but a requirement for decision making. The tendency of current models to generate plausible sounding but ungrounded numerical estimates, a phenomenon one might term "numerical hallucination," poses significant risks for deployment in high stakes environments.

Furthermore, the distinction between enumeration and counting highlighted in this work suggests a broader architectural question. Current MLLMs treat counting as a regression or generation task, producing a number based on statistical regularities in training data. EC-Bench demonstrates that reliable counting requires explicit, symbolic enumeration processes that track individual instances through time. This suggests that future architectures may need to incorporate explicit memory mechanisms, temporal state machines, or neurosymbolic components that maintain instance level representations across video durations.

Conclusion

EC-Bench establishes a new standard for evaluating quantitative reasoning in long form video, moving beyond aggregate accuracy metrics to examine the component processes underlying counting behavior. The benchmark reveals that current MLLMs face fundamental limitations in temporal grounding and instance enumeration that cannot be resolved through scale alone. These findings suggest that the path to robust long video understanding requires architectural innovations capable of maintaining object permanence and temporal coherence across extended contexts.

Looking forward, several questions remain open. Can enumeration ability be improved through explicit training on temporal tracking objectives, or does it require fundamentally different architectural priors? How do these limitations generalize to other forms of quantitative reasoning in video, such as measuring durations or detecting periodicities? EC-Bench provides the diagnostic tools necessary to pursue these questions, offering a framework where models must demonstrate not just the final answer, but the reasoning process that produced it. Until MLLMs can reliably enumerate what they see, their understanding of long videos will remain superficial, no matter how impressive their generated prose.