Why Text-to-Video Models Cannot Count, and What to Do About It

Ask a modern text-to-video model for "three cats playfully chasing each other through the house" and you will likely get two cats, or four, or a blurry pile that resists enumeration. The visual fidelity is often excellent. The motion is smooth. The semantics are roughly correct. But the number is wrong. This failure is surprisingly stubborn, and it points to something structural about how diffusion transformers represent the world.

A recent paper, "When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models" by Sun, Chen, and colleagues from Huazhong University of Science and Technology, takes this problem seriously and offers a diagnosis along with a training-free remedy called NUMINA. I want to walk through what they found, why it matters, and where I think this line of work leads.

The Diagnosis: Numerals Do Not Localize

The most useful contribution of the paper is not the method itself but the analysis of why counting fails. The authors probe Wan2.1-1.3B, a widely used open T2V model, and isolate two distinct causes.

The first is semantic weakness. When you visualize cross-attention maps for different token types in a prompt like "Three men in black jackets are running in the park," the nouns ("men," "jackets"), the adjective ("black"), and the verb ("are running") all produce sharp, spatially localized attention patterns. Each word grabs onto specific regions of the latent. The numeral "Three," by contrast, produces a diffuse, spread-out response. It does not bind to any particular set of regions. The model has learned what a man looks like and what running looks like, but "three" never acquired a grounded visual meaning during training.

This makes intuitive sense if you think about how these models learn. A noun co-occurs with a consistent visual signature across millions of captioned clips. A numeral co-occurs with wildly varying visual scenes; "three" appears with three cats, three buildings, three abstract shapes, three anything. There is no stable visual feature for the model to attach the token to. Numerals are inherently relational and combinatorial, which is exactly the kind of structure that gradient descent on next-frame prediction does not naturally extract.

The second cause is instance ambiguity. DiT-based video models operate in a heavily downsampled spatiotemporal latent space. That compression is what makes them efficient, but it also smears individual object representations together. Even if the model "wanted" to allocate exactly three separable regions, the latent resolution makes clean instance separation difficult. Counting requires discrete, countable entities, and the latent space is built for continuous appearance, not discrete enumeration.

These two factors compound. Weak numeral grounding means the count constraint is barely encoded, and poor instance separability means that even a correctly encoded count cannot be reliably realized.

The Fix: Identify, Then Guide

NUMINA is a training-free, identify-then-guide framework. This design choice is pragmatic. Retraining a 14B-parameter video model with precise count annotations would require building a large dataset with reliable object-count labels, which is expensive and error-prone, and would risk degrading other attributes like motion coherence. So the authors intervene at inference time instead.

The framework operates in two phases:

• Identify. Early in the denoising process, NUMINA inspects the self- and cross-attention maps to estimate the current latent layout, meaning the spatial distribution of object-related activations. Crucially, not all attention heads are equally informative for counting. The method applies a dynamic selection of attention heads using an object discriminability criterion, keeping the heads that best separate instances. A cluster-based algorithm then turns those activations into a discrete segmentation, producing a countable layout that can be compared against the prompt's numeral.

• Guide. If the identified count does not match the prompt, NUMINA refines the layout conservatively. It adds or removes instances while heuristically respecting spatial relationships between existing objects, then modulates cross-attention during regeneration to steer the model toward the corrected layout.

The conservatism matters. A naive intervention that aggressively rewrites the latent would damage appearance, motion, and temporal coherence. By treating the layout adjustment as a light, structure-level edit and letting the model's own generative prior fill in the rest, NUMINA preserves what the base model does well.

The Results, and How to Read Them

To evaluate this, the authors introduce CountBench, 210 prompts spanning counts from 1 to 8 and scenes with 1 to 3 object categories. On this benchmark, NUMINA improves counting accuracy by:

• 7.4% on Wan2.1-1.3B
• 4.9% on the 5B model
• 5.5% on the 14B model

Alongside the counting gains, CLIP alignment scores increased across baselines, and temporal consistency was maintained. The method also composes with inference acceleration techniques like EasyCache, which is important because attention-map analysis adds overhead that you would want to claw back.

Two observations are worth dwelling on. First, the gains are real but modest in absolute terms. A 5 to 7 percent improvement leaves a substantial residual error, which tells us that inference-time guidance addresses the symptom (wrong layout at generation time) more than the disease (numerals are not grounded in the representation). Second, the fact that CLIP score also rises is a quiet but meaningful result. It suggests that getting the count right is not a trade-off against overall text-video alignment; rather, correct instance counts produce cleaner, less cluttered scenes that improve global alignment. Counting and quality are aligned objectives, not competing ones.

My Take: Counting Exposes a Deeper Representational Gap

What I find most interesting here is what counting reveals about diffusion transformers as a class of models. These systems are extraordinary interpolators of appearance and motion, but they lack an explicit notion of object identity persisting as a discrete unit. Counting is the cleanest possible test of object-individuation, because it forces the model to commit to a specific cardinality. The diffuse attention on numerals is a fingerprint of a representation that thinks in terms of texture and region, not in terms of countable things.

NUMINA works by externally imposing the discrete structure that the model fails to maintain internally. This is the same pattern we have seen across the broader field: layout-guided image generation, attention re-weighting methods like attend-and-excite for missing objects, and bounding-box conditioning all inject discrete structure that the base generative prior does not reliably produce on its own. NUMINA is the video-domain, training-free instance of this family, with the added difficulty that the discrete entities must remain consistent across frames.

The honest limitation is that this is a patch over a missing capability. The clustering-based segmentation depends on the base model's latent already being separable enough to cluster, which is precisely the property the paper identifies as weak. For high counts (the benchmark goes to 8) or visually overlapping instances, I would expect the identify step to degrade, and the method cannot conjure separability that the latent does not contain.

Where This Goes Next

I will repeat the prediction I made earlier in shorter form. Over the next two to three years, I expect count and spatial constraints to become first-class conditioning signals rather than emergent behaviors we hope for. The natural architectural response to the instance-ambiguity problem is explicit instance tokens: latent slots, in the spirit of object-centric and slot-attention models, where "three cats" binds to three tracked slots that persist across frames. That would address both the grounding problem (numerals condition slot allocation directly) and the separability problem (slots are discrete by construction).

If that happens, numerical fidelity becomes a standard benchmark axis alongside FVD and CLIP score, and CountBench-style evaluations become routine. NUMINA is valuable precisely because it works today, without retraining, and because its analysis makes the path forward legible.

The open question I keep returning to: can a model learn discrete object-individuation purely from video prediction at scale, or is some inductive bias toward countable entities necessary? Counting may be the cheap, measurable proxy that finally forces this question into the open.