The Protocol Problem: Why MLLM Classification Metrics Are Fundamentally Unstable

Recent literature presents contradictory portraits of Multimodal Large Language Model (MLLM) capabilities. Some studies report that models like GPT-4V or LLaVA lag far behind supervised vision models on standard classification benchmarks. Others claim performance parity with Vision-Language Models such as CLIP. This discrepancy is not merely academic. In safety critical applications like medical diagnostics or species identification, understanding true model capabilities determines deployment decisions.

The source of these contradictions lies not in the models themselves, but in how we measure them. In their recent study, "Multimodal Large Language Models as Image Classifiers," Kisel and colleagues from the Visual Recognition Group at Czech Technical University demonstrate that evaluation protocol choices create variance comparable to, or exceeding, the architectural differences between models. Their work exposes a measurement crisis in multimodal AI.

The Three Protocols and Their Pathologies

The paper identifies three distinct evaluation paradigms. First, Open World (OW) allows models to generate free form descriptions later mapped to class labels. Second, Multiple Choice (MC) forces selection from predefined distractors. Third, Closed World (CW) requires selection from the full candidate set.

Each protocol carries specific biases. The OW approach, most aligned with real world usage, historically underperformed due to crude string matching heuristics. Kisel et al. show that adopting embedding space nearest neighbor mapping transforms OW from the weakest to the strongest protocol for half of tested models. This inversion suggests prior comparisons fundamentally mischaracterized MLLM capabilities.

The MC protocol, ubiquitous in leaderboard evaluations, systematically inflates performance. By limiting choices to small candidate sets with weak distractors, MC creates an artificial ceiling that masks true error modes. When the authors introduce semantically challenging distractors, accuracy drops substantially, particularly for newer models. The protocol effectively provides multiple choice questions where the correct answer is obvious by elimination rather than visual understanding.

Closed World evaluation faces the Out Of Prompt (OOP) problem. When presented with full class vocabularies (up to 1000 classes for ImageNet), MLLMs frequently generate valid labels absent from the provided list. Previous work discarded these predictions as errors. The authors propose CW+, which resolves OOP responses via the same embedding based mapping used in OW. This simple fix eliminates the primary objection to CW evaluation, rendering MC protocols unnecessary except for token constrained models.

Ground Truth and the Annotation Bottleneck

Beyond protocol engineering, the paper confronts dataset quality. ImageNet 1k, the de facto standard for visual recognition, contains approximately 20% label errors in its validation set, with 15 to 21% of images containing multiple objects from different classes. These limitations are well documented, yet they persist as the foundation for comparing MLLMs against supervised baselines.

Kisel et al. introduce