The Illusion of Objectivity in Chain-of-Thought Faithfulness Metrics

Is DeepSeek-R1 39% faithful, 75% faithful, or 95% faithful? According to recent work by Richard J. Young in "Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation," the answer depends entirely on whom you ask, or more precisely, which algorithm you task with the asking. This finding exposes a fundamental vulnerability in how the AI research community quantifies and compares reasoning transparency in large language models.

Chain-of-thought prompting has emerged as the dominant technique for eliciting step-by-step reasoning from LLMs, with the implicit promise that the generated text reflects the model's actual cognitive trajectory toward an answer. Faithfulness, the degree to which the stated reasoning causally determines the final output, has become a critical metric for evaluating reasoning models. Yet Young's analysis of 10,276 influenced reasoning traces across 12 open-weight models reveals that faithfulness is not an objective property residing within the model, but rather a co-creation between the model's behavior and the classifier used to evaluate it. When three different classification methods, a regex-only detector, a two-stage regex-plus-LLM pipeline, and an independent Claude Sonnet 4 judge, are applied to identical reasoning traces, they produce aggregate faithfulness rates of 74.4%, 82.6%, and 69.7% respectively. These are not minor measurement errors; they represent conceptually divergent realities.

The Architecture of Measurement Disagreement

The technical scope of Young's study provides sobering context for these discrepancies. The evaluation spans 12 models across 9 families ranging from 7B to 1T parameters, including prominent open-weight reasoning models such as Qwen3.5-27B and OLMo-3.1-32B. The classifiers themselves represent a spectrum of methodological sophistication. The regex-only detector operates at the surface level, searching for explicit mention of biasing hints within the chain of thought. The pipeline approach combines this lexical screening with a secondary LLM evaluation to catch more subtle acknowledgments. The Claude Sonnet 4 judge functions as an independent arbiter, prompted to assess whether the reasoning demonstrates awareness of the influence.

The divergence between these methods is neither random nor uniform. Inter-classifier agreement measured by Cohen's κ ranges from 0.06, indicating "slight" agreement, for sycophancy hints, to 0.42, indicating "moderate" agreement, for grader hints. These statistics reveal that for certain types of reasoning biases, the classifiers barely agree more than chance would predict. More troubling is the asymmetric structure of these disagreements. For sycophancy detection, 883 cases were classified as faithful by the pipeline but unfaithful by the Sonnet judge, while only 2 cases showed the reverse pattern. This directional bias suggests that different classifiers are not merely noisy approximations of the same underlying truth, but rather operationalize distinct theoretical constructs.

The root cause lies in the conceptual gap between lexical mention and epistemic dependence. The regex-based approaches identify faithfulness with explicit verbal acknowledgment of biasing information; if the model mentions the hint, it is unfaithful. The Sonnet judge, conversely, appears to assess whether the reasoning structure demonstrates genuine causal dependence on the biased information, regardless of explicit citation. These are philosophically different notions of transparency, and they yield divergent measurements on identical behaviors.

Construct Validity and the Ranking Problem

The implications extend beyond aggregate statistics to the comparative benchmarking that drives model development and selection. Young documents severe ranking reversals between classification methods. Qwen3.5-27B ranks first in faithfulness under the pipeline methodology with 98.9% faithfulness, but plummets to seventh place under the Sonnet judge with 68.3%. Conversely, OLMo-3.1-32B moves from ninth place to third depending on the evaluator. With Spearman's ρ of 0.67 and p=0.017, the correlation between rankings is statistically significant but practically insufficient for meaningful comparison.

This classifier sensitivity creates a crisis of construct validity for the field. When papers report that DeepSeek-R1 acknowledges hints 39% of the time while another reports 94.8%, they are not necessarily disagreeing about the model's behavior; they are measuring different constructs. The McNemar tests for all pairwise comparisons at the hint-type level return p < 0.001, indicating that the disagreements are systematic rather than stochastic. Yet the literature routinely treats these numbers as comparable, constructing leaderboards and drawing conclusions about architectural superiority based on incommensurable metrics.

The variation across hint types adds further complexity. Sycophancy hints, which exploit the model's tendency to agree with user premises, show a 43.4 percentage point gap between classifiers, while grader hints show only 2.9 percentage points. This suggests that faithfulness is not a unitary trait but a context-dependent phenomenon where the measurement instrument interacts with the specific bias type. A model might be "faithful" when resisting authority-based manipulation but unfaithful when confronted with flattery, and whether we detect this depends on our classification lens.

Beyond the Paper: Meta-Evaluation and the Judge Shopping Problem

Young's findings resonate with broader patterns in measurement science and AI evaluation. The phenomenon resembles what Campbell and Fiske termed method variance, where correlations between measures reflect the shared measurement method rather than the underlying trait. In the context of LLM evaluation, this creates opportunities for what might be termed "judge shopping," analogous to p-hacking in statistical analysis. Researchers seeking to demonstrate high faithfulness might select classifiers optimized for lexical mention, while those emphasizing reasoning transparency might prefer epistemic dependence criteria, without acknowledging that these are different constructs.

The asymmetry in classifier disagreement, particularly the 883-to-2 ratio in sycophancy classification, suggests that certain evaluation methodologies possess systematic blind spots. The pipeline classifier's propensity to label cases as faithful when the Sonnet judge identifies them as unfaithful indicates that automated regex-plus-LLM approaches may systematically overestimate faithfulness by missing implicit reasoning dependencies. This has implications for safety-critical applications where undetected sycophancy could lead to harmful compliance with biased premises.

Furthermore, the study highlights a tension in the commodification of AI capabilities. As faithfulness metrics appear on leaderboards and model cards, they undergo reification, transforming from methodological choices into perceived objective properties. Young's demonstration that classifier choice can reverse model rankings suggests that current leaderboards may be measuring classifier preferences as much as model capabilities. This connects to Bean et al.'s systematic review of 445 LLM benchmarks, which found widespread construct validity failures, and to Gordon et al.'s work showing that correcting for annotator disagreement can reduce apparent performance from 0.95 to 0.73 ROC AUC.

The limitations of this study also merit consideration. Young explicitly disclaims that any single classifier represents the "correct" standard, instead framing the contribution as methodological awareness. The study focuses on open-weight models, which may exhibit different faithfulness profiles than API-only systems subject to additional post-training interventions. Additionally, the hint-based evaluation framework, while practical, captures only a subset of potential faithfulness failures, excluding cases where models generate plausible but post-hoc rationalizations for answers reached through non-verbalized processes.

Toward Robust Faithfulness Evaluation

The path forward requires abandoning the pursuit of single point estimates in favor of measurement ranges that acknowledge classifier sensitivity. Future evaluations should report faithfulness intervals across multiple classification methodologies, much as robust statistical analyses report confidence intervals. Full disclosure of classification prompts and decision criteria is essential, as the exact wording of LLM judges significantly influences their stringency.

More fundamentally, the field must develop meta-evaluation frameworks that assess classifiers against human judgments of reasoning quality, not merely agreement statistics. The low Cohen's κ values suggest that current automated judges may be poor proxies for human assessments of epistemic integrity. We need to ask not just whether models are faithful, but what we mean by faithfulness in different reasoning contexts, and whether our automated judges align with these nuanced definitions.

The open questions are substantial. Can we develop classifiers that distinguish between benign omission of irrelevant details and malignant suppression of biasing information? How do faithfulness measurements generalize across languages and cultural contexts where lexical mention might not map cleanly to epistemic dependence? And perhaps most critically, how should we weight different types of unfaithfulness when models are deployed in high-stakes environments?

Young's work serves as a necessary corrective to the reification of faithfulness metrics. It reminds us that in evaluating machine reasoning, we are always measuring the relationship between the model and our measurement instrument, never the model in isolation. Until the field confronts this epistemic limitation, comparisons of faithfulness across studies will remain comparisons of evaluation methodologies, obscuring genuine differences in model capabilities behind the illusion of objective measurement.