When Explanations Mislead: Reevaluating Shapley Values Through Human Behavior

Introduction

In high stakes operational environments, from fraud detection to credit underwriting, machine learning models do not operate in isolation. Human analysts review algorithmic outputs, weighing predictions against institutional knowledge and contextual nuance before rendering final decisions. This human in the loop workflow has driven the widespread adoption of explainable AI techniques, particularly Shapley values, which decompose model predictions into feature level contributions grounded in cooperative game theory. Yet a troubling disconnect has emerged between the mathematical elegance of these methods and their practical utility in decision support.

A recent comprehensive audit, Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings, reveals that our standard evaluation paradigms may not merely be incomplete, but actively misleading. Conducted across 3,735 real case reviews involving 37 professional fraud analysts, the study demonstrates that quantitative optimization of Shapley formulations for sparsity and faithfulness bears no correlation with human perceived clarity or decision quality. More concerningly, explanations consistently increased analyst confidence without improving objective accuracy, documenting a measurable form of automation bias in production grade environments.

The Fragmented Semantics of Shapley Values

Shapley values provide an axiomatic framework for attributing a model's output to individual input features by computing each feature's marginal contribution across all possible coalitions. Formally, for a model $f$ and input $\boldsymbol{x}$, the value function $v_{\boldsymbol{x}}(S)$ defines the expected output when only features in subset $S$ are observed, marginalizing over absent features according to a background distribution $p(\cdot)$. The Shapley value $\phi_i$ aggregates these marginal contributions with specific weighting factors ensuring efficiency, symmetry, and dummy axioms.

However, the operationalization of Equation (1) in the paper, which defines how absent features are handled, has fragmented into competing formulations with divergent semantic interpretations. The authors isolate eight distinct variants: Baseline Zero, Baseline Mean, Marginal, Uniform, Joint Marginal, Filtered Conditional, Conditional, and Counterfactual. Each embodies a different philosophical stance on feature absence, ranging from replacing missing values with dataset means to sampling from conditional distributions or counterfactual baselines.

To enable fair comparison, the researchers implement a unified amortized framework that eliminates algorithmic confounders such as sampling variance or approximation errors. This technical isolation is crucial; previous comparisons often conflated implementation choices with definitional semantics, obscuring whether observed differences stemmed from theoretical foundations or computational approximations. By amortizing the computation under identical low latency constraints relevant to financial and healthcare deployments, the study isolates the pure semantic effect of each formulation.

The Metric Utility Gap

The central empirical contribution of the work lies in documenting the decoupling between quantitative proxies and human utility. Standard XAI evaluation relies on functional metrics such as sparsity, which measures the concentration of attribution mass on few features, and faithfulness, typically assessed via insertion AUC curves that track prediction recovery as features are added in order of attribution importance. These metrics assume that sparse, faithful explanations better serve human decision makers.

The results refute this assumption. Across four risk datasets and the live fraud detection environment, the eight Shapley formulations exhibit systematic variation in sparsity and insertion AUC scores. Yet when 37 professional analysts reviewed 3,735 cases, these quantitative rankings showed zero alignment with subjective clarity ratings or objective decision performance. Formulations optimized for high insertion AUC, for instance, did not yield higher analyst accuracy or perceived comprehensibility. The trade off curves between sparsity and faithfulness, visualized in Figure 1 of the paper, occupied entirely different regions than the human rated clarity confidence trade offs, suggesting that current benchmark optimization targets a latent space unrelated to human cognitive utility.

This misalignment exposes a fundamental methodological vulnerability in the XAI literature. We have been treating mathematical desiderata, such as local accuracy or consistency, as sufficient proxies for explanatory value without empirical validation that these properties translate into improved human reasoning. The study provides evidence that quantitative benchmarks are insufficient for predicting downstream human impact, rendering them potentially hazardous as sole optimization objectives.

Automation Bias in High Stakes Workflows

Perhaps the most consequential finding concerns the psychological impact of explanations on analyst judgment. While explanations did not improve objective decision accuracy across any formulation, they consistently increased decision confidence as measured by odds ratios. This divergence between calibration and performance signals a critical failure mode: automation bias induced by explanatory interfaces.

In high stakes domains where predictions carry direct financial or human consequences, elevated confidence without commensurate accuracy enhancement is not a neutral outcome; it is a risk amplification mechanism. Analysts presented with Shapley attributions reported higher clarity perceptions and greater certainty in their judgments, yet their error rates remained statistically indistinguishable from unassisted baselines. This suggests that feature attribution visualizations may function as rhetorical devices that lend unwarranted credibility to model outputs, encouraging complacent acceptance rather than critical evaluation.

The study notes that this effect persisted across all eight semantic variants, indicating that the automation bias risk is not merely an artifact of specific implementation choices, such as baseline selection, but potentially inherent to the current paradigm of additive feature attribution. When explanations are deployed in operational risk workflows under time pressure and regulatory scrutiny, the confidence boosting effect may override the very oversight functions these tools are designed to support.

Original Insights: Beyond Proxy Optimization

This audit illuminates a broader pathology in contemporary XAI research: the optimization trap. By treating interpretability as a function to be maximized against internal mathematical consistency checks, we have created a closed loop of technical refinement that bypasses the messy reality of human cognition. The Shapley framework, despite its axiomatic foundations, appears to optimize for computational convenience rather than epistemic support.

I would argue that the findings suggest a category error in how we conceptualize explanation quality. Current approaches treat explanations as compression artifacts, reducing high dimensional model behavior into sparse feature lists. Yet human decision making in complex domains relies on causal narratives, counterfactual reasoning, and structural understanding of data generating processes, none of which are necessarily captured by marginal contribution scores. The null result regarding accuracy improvement suggests that Shapley values may provide information that is statistically sufficient but cognitively insufficient for human operators.

Furthermore, the consistent elevation of confidence across all formulations raises questions about the interface design of XAI systems. Perhaps the issue lies not in the attribution algorithm but in the presentation modality, which may implicitly signal authority or completeness. The study's release of 3,735 granular human AI interaction measurements provides a valuable resource for investigating whether alternative visualization strategies or interactive explanation modes can mitigate automation bias while preserving the benefits of algorithmic transparency.

Conclusion

The work of Oliveira e Silva et al. serves as a necessary corrective to the unchecked proliferation of Shapley based explanations in production systems. Their evidence suggests that we must fundamentally rethink XAI evaluation as a behavioral science rather than a mathematical optimization problem. The field requires new benchmarks grounded in cognitive psychology and decision theory, measuring not just faithfulness to the model but calibration of the human model partnership.

Moving forward, critical questions remain unanswered. Can we design attribution methods that specifically target human accuracy rather than confidence? How do we validate explanation utility without expensive human trials for every deployment? And perhaps most importantly, should high stakes domains adopt a presumption of non explanation for certain model classes until human benefit is demonstrated? The audit provides compelling evidence that current practices are insufficient; the burden of proof now lies with practitioners to demonstrate that their explanatory tools enhance, rather than merely decorate, human judgment.