Why Vision Language Models Fail at Reading Human Emotions

Vision language models have become remarkably good at a wide range of visual tasks. They can describe scenes, ground complex textual queries, read charts, and reason about spatial relationships. So it is striking that when you point them at a video of a human face and ask a simple question, "what emotion is this person feeling?", even the most capable contemporary models stumble. They often perform no better than, and sometimes worse than, much smaller vision-only classifiers trained specifically for the task.

The paper "Why Do Vision Language Models Struggle To Recognize Human Emotions?" by Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara, and Steven McDonagh from the University of Edinburgh takes this failure seriously and traces it to two compounding root causes. I want to walk through their argument, add some of my own interpretation, and then discuss where I think the next experiments should go.

The Symptom: A Persistent Performance Gap

The authors frame dynamic facial expression recognition (DFER) as a probe task. It is a good choice because it stresses two capabilities at once: fine-grained spatial sensitivity to subtle facial muscle movements, and temporal fidelity to track how those movements evolve over time. If a model is going to fail at temporal affective reasoning, this is where the cracks will show.

The qualitative failures in the paper are telling. Gemini 2.5 Flash, Qwen2.5-VL, and even EmotionQwen, a model tuned for affect, misclassify "Happiness" as "Sadness," "Neutral," or "Disgust," and confuse "Anger" with "Anxiety" or "Neutral." These are not random errors. A brief smile gets overlooked. A momentary tensing of the face gets smoothed into a neutral reading. The pattern that emerges is one of collapse toward common, low-arousal categories, particularly neutral.

This is not a question of the models lacking world knowledge about emotions. They can define and reason about emotional states fluently in text. The failure is in perception and temporal binding, not in semantics.

Root Cause One: The Long Tail, Amplified

The first culprit is the long-tailed distribution of emotional categories, and the way web-scale pretraining magnifies it.

Emotion datasets are naturally skewed. Neutral and a few high-frequency states dominate, while consequential but rare emotions appear infrequently. This alone would be a problem. But the paper makes a sharper point. VLMs are trained by aligning visual features with text embeddings drawn from web alt-text and captions during contrastive pretraining. The consequence is that the lexical frequency of an emotion word in human-generated text directly shapes how densely that concept is represented in the multimodal latent space.

Because the actual pretraining corpora are proprietary, the authors cannot measure emotion class prevalence directly. Instead they use a clever proxy: historical lexical frequencies of emotion terms from Google Books Ngrams. They find a pronounced long tail in how often emotion words appear in human writing, and they show that class sparsity is inversely associated with per-class accuracy. The rarer the emotion, the larger the error rate.

I find this argument convincing because it connects two distributions that are usually treated separately. The dataset long tail and the language long tail are not independent problems that happen to coincide; they are the same bias entering the model through two doors. The contrastive objective effectively bakes in a prior that says common emotions are likely and rare ones are not. When the model is uncertain, it falls back on that prior, which is exactly why rare emotions collapse into neutral.

The authors propose alternative sampling strategies that avoid favoring common concepts, and they validate the hypothesis through controlled dataset balancing. This is the part of the analysis that moves from correlation to something closer to causal evidence: if you balance the data, the head-class bias should diminish.

Root Cause Two: Sparse Sampling Versus Fleeting Signals

The second cause is more architectural. VLMs have fixed context windows and finite token budgets. Video gets expensive fast, so the standard practice is to sample a sparse set of keyframes, what the authors aptly call a "bag-of-frames" strategy. This sampling is often order-agnostic, which already disrupts the temporal dynamics that emotion depends on.

The deeper problem is a mismatch of timescales. Micro-expressions, the brief and often involuntary facial movements that carry the most diagnostic affective signal, last only 0.25 to 0.5 seconds. Uniform sparse sampling will frequently miss them entirely. You can sample ten frames from a clip and never land on the quarter-second window where the true emotion surfaces.

The authors document a quasi-bell-shaped relationship between performance and the number of frames. Too few frames and you miss the signal. Too many and you dilute the relevant cues with redundant frames, while the model's ability to retain and prioritize information degrades as context length grows, a limitation already documented in long-context studies. There is a sweet spot, but it is narrow and fragile.

Their diagnostic intervention is elegant. Rather than throwing more frames at the model, they use a multi-stage context enrichment strategy. The "in-between" frames that would otherwise be dropped are first converted into natural language summaries. This enriched text is fed to the VLM alongside the sparse visual keyframes. The idea is to preserve the emotional trajectory without paying the full visual token cost, avoiding attentional dilution while keeping the temporal story intact.

This reframing is important. The bottleneck is not that the model cannot process emotion; it is that the relevant evidence rarely makes it into the context in a usable form.

My Take: Selection, Not Scale

What I find most useful about this paper is that it reframes the emotion recognition bottleneck as a selection problem rather than a scale problem. The instinct in the field is to assume that bigger models and longer context windows will eventually solve perception failures. But if the discriminative signal lives in a quarter-second micro-expression, uniform sampling at any reasonable frame budget will miss it most of the time. Scaling context helps only if you also fix what you put into it.

This suggests a concrete next step that I want to pursue: train a lightweight temporal adapter that scores frames by facial motion energy, then route only the high-saliency windows to dense sampling while summarizing the remaining frames as text, in the spirit of the paper's enrichment strategy. If micro-expression peaks really are the discriminative signal, then adaptive, saliency-driven sampling should outperform uniform keyframe sampling without inflating the token budget. The motion-energy scorer is cheap, runs once per clip, and turns a fixed budget into a budget spent where it matters.

There is a natural synergy between the two root causes here. Adaptive sampling addresses the temporal failure, but it may also help the long-tail problem indirectly. Rare emotions are often the ones with the most distinctive transient expressions. If you reliably capture those peaks, you give the model a stronger visual signal to override its frequency prior. The two fixes could compound in the right direction rather than the wrong one.

Limitations and Open Questions

A few caveats deserve honesty. The Google Books Ngrams proxy is a correlative indicator, not a direct measurement of pretraining frequency. The relationship between book-text word frequency and web alt-text frequency is plausible but unverified. The text-summarization enrichment strategy also introduces a dependency on whatever model generates the summaries, which could inject its own biases or hallucinations into the temporal description.

Several questions remain open. Does the head-class bias persist after instruction tuning, or is it primarily a contrastive pretraining artifact? Can a motion-energy scorer generalize across cultures and lighting conditions where micro-expressions present differently? And more broadly, is natural language the right intermediate representation for temporal affect, or does converting motion into words discard the very subtlety that makes emotion hard?

The paper does not claim to have solved emotion recognition. What it does, valuably, is show that the failure is structured and diagnosable. That is the precondition for fixing it, and it points the field toward smarter selection rather than simply more data.