Beyond Accuracy: Why Explanation Consistency Matters More Than Localization in Medical AI

The Deployment Gap Between Performance and Trust

Modern convolutional neural networks for chest X-ray analysis now routinely achieve area under the receiver operating characteristic curve values exceeding 0.99 for pneumonia detection. These numbers suggest near perfect discriminative capability, yet translation into clinical practice remains hesitant. The disconnect stems from a fundamental epistemological problem: high AUC certifies that a model correctly ranks pathological cases above normal ones, but it reveals nothing about the visual features driving that ranking. A model might achieve stellar classification performance by detecting equipment markers, patient positioning artifacts, or pacemaker presence rather than parenchymal opacities. This "Clever Hans" phenomenon, where models exploit dataset specific shortcuts imperceptible to radiologists, creates a verification gap that threatens patient safety.

The paper "Quantifying Explanation Consistency: The C-Score Metric for CAM-Based Explainability in Medical Image Classification" by Elangovan and Ting addresses this gap not by asking whether explanations are correct, but whether they are consistent. The authors argue that intra-class explanation reproducibility is a necessary precondition for clinical interpretability, and they propose a metric that quantifies this property without requiring expensive radiologist annotations.

The Limitations of Current Evaluation Frameworks

Class Activation Mapping methods were introduced to render CNN spatial reasoning visible and auditable. However, foundational studies have revealed that these methods suffer from reliability failures invisible to classification metrics. Model parameter randomization experiments demonstrate that several popular techniques produce nearly identical heatmaps for fully trained models and randomly initialized ones. Input invariance problems show that constant pixel shifts, which do not affect predictions, can completely alter saliency attributions. These failures suggest that current CAM methods may not measure learned reasoning at all.

Existing evaluation frameworks compound these problems by focusing exclusively on localization fidelity. Metrics like the pointing game or intersection over union with bounding boxes compare CAM predicted attention regions to ground truth annotations. While clinically relevant, these approaches require expensive per image annotation unavailable at scale, cannot be applied across training checkpoints, and conflate attribution quality with model behavior on corrupted data. Most critically, they fail to address intra-class consistency. Current metrics ask whether the model attends to the correct spatial region on a per image basis. The fundamentally different clinical question, whether the model applies the same visual reasoning strategy across different patients with the same diagnosis, remains unanswered.

Elangovan and Ting identify this as the central reliability problem. Consistent wrong focus is addressable; clinicians can learn to interpret and mentally correct for systematic biases. Inconsistent focus is unpredictable. A physician cannot develop a reliable mental model of a system whose attention is arbitrarily variable across patients.

The C-Score Methodology and Mechanisms of Dissociation

The proposed Consistency Score, or C-Score, quantifies intra-class explanation reproducibility via confidence weighted, intensity emphasised pairwise soft intersection over union across correctly classified instances. Unlike localization metrics, C-Score requires no ground truth annotations. It operates by comparing CAM heatmaps across all correctly classified instances of a given class, measuring whether the model attends to similar anatomical regions across different patients.

The authors evaluate six CAM techniques, GradCAM, GradCAM++, LayerCAM, EigenCAM, ScoreCAM, and MS-GradCAM++, across three CNN architectures, DenseNet201, InceptionV3, and ResNet50V2, over thirty training epochs on the Kermany chest X-ray dataset. This longitudinal design, covering both transfer learning and fine tuning phases, enables detection of temporal dynamics invisible to single checkpoint evaluations.

Their analysis reveals three distinct mechanisms of AUC consistency dissociation, phenomena where classification performance and explanation consistency diverge. First, threshold mediated gold list collapse occurs when high confidence correct predictions mask underlying attribution instability. Second, technique specific attribution collapse at peak AUC shows that some CAM methods degenerate precisely when classification performance is highest. Third, class level consistency masking in global aggregation demonstrates that aggregate metrics can hide systematic inconsistency in specific pathological classes.

Perhaps most clinically relevant is the early warning capability demonstrated for ScoreCAM on ResNet50V2. The C-Score detects explanation deterioration one full checkpoint before catastrophic AUC collapse. This suggests that explanation consistency serves as a leading indicator of model instability, providing a temporal window for intervention before predictive performance degrades.

Clinical Implications and Unresolved Tensions

The C-Score represents a significant conceptual shift in medical AI evaluation. By prioritizing consistency over correctness, it acknowledges the reality that deep learning models lack causal common sense and may diverge from human visual search patterns. Ensuring these divergent strategies remain stable across patient populations becomes a necessary safeguard against spurious correlations.

However, this approach introduces its own limitations that warrant honest discussion. The C-Score assumes correctly classified instances are the appropriate substrate for consistency measurement, yet this excludes the very cases where explanation quality matters most. A model might be highly consistent on correct predictions but wildly inconsistent on errors, precisely the scenario where clinicians need interpretability to understand failure modes. Additionally, the metric cannot distinguish between consistent right and consistent wrong explanations. A model systematically attending to the wrong anatomical region would score highly on C-Score while remaining clinically dangerous.

The confidence weighting scheme, while mathematically elegant, may also create blind spots. If a model is confidently wrong in a consistent manner, the C-Score would indicate high explanation quality. This suggests the metric should function not as a standalone deployment criterion but as one component