Beyond Verbalization: Reassessing Chain-of-Thought Faithfulness Through the Lens of Compression and Causal Mediation

The prevailing narrative in large language model interpretability has taken a skeptical turn regarding Chain-of-Thought (CoT) reasoning. Recent studies using hint injection methods have painted a troubling picture: models appear to ignore biasing features in their reasoning traces, suggesting CoT serves as post-hoc rationalization rather than genuine explanation. However, a careful new analysis by Kerem Zaman and Shashank Srivastava in their paper "Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization" challenges this consensus. Their work reveals that what appears to be unfaithfulness may instead reflect the fundamental compression inherent in transforming distributed transformer computation into linear natural language narratives.

The Biasing Features Paradox and Metric Disagreement

The dominant methodology for assessing CoT faithfulness, termed Biasing Features, operates through a straightforward mechanism. Researchers inject a hint into the prompt that biases the model toward a specific answer, then evaluate whether the generated CoT explicitly mentions this hint. If the model changes its prediction to match the hint but fails to verbalize it in the reasoning trace, the CoT is labeled unfaithful. This approach underlies recent claims that state-of-the-art models produce highly unreliable explanations.

Yet Zaman and Srivastava identify a critical flaw in this reasoning. When they evaluated the same CoT traces using alternative faithfulness metrics, such as Filler Tokens (which corrupts the CoT and measures prediction changes) and FUR, or Faithfulness through Unlearning Reasoning Steps (which tests whether removing individual reasoning steps alters the output), they found striking discrepancies. For some models, including Llama-3 and Gemma-3 variants, over 50% of CoTs flagged as unfaithful by Biasing Features were classified as faithful under these alternative evaluation schemes.

This metric disagreement raises profound questions about what we mean by "faithfulness" in the context of neural network interpretability. Biasing Features assumes a specific transparency relationship: if an input feature causally influences the output, that feature must appear explicitly in the explanation. However, this assumption conflates two distinct phenomena: unfaithfulness, where the explanation misrepresents the underlying computation, and incompleteness, where the explanation provides a lossy but accurate summary of the reasoning process. The distributed, parallel nature of transformer inference does not naturally map to the sequential, selective format of natural language explanation. Requiring explicit verbalization of every causal influence ignores the compression necessary to convert high-dimensional activation patterns into human-readable text.

Inference Budget, Compression, and the faithful@k Analysis

To test whether apparent unfaithfulness stems from compression constraints rather than misalignment, the authors introduce a novel metric: faithful@k. This metric examines whether increasing the inference-time token budget improves hint verbalization rates. The results are illuminating. Under standard token constraints, many CoTs fail to mention injected hints, contributing to the narrative of widespread unfaithfulness. However, when the models are allowed larger generation budgets, the probability of obtaining at least one hint-verbalizing CoT increases dramatically, reaching up to 90% in certain settings.

This finding suggests that much of what Biasing Features captures is not deception or post-hoc rationalization, but rather the economic constraints of explanation generation. CoT serves as a form of working memory for transformers, but like human working memory, it faces capacity limitations. When forced to compress complex, multi-hop reasoning into a limited number of tokens, models must select which aspects of their computation to verbalize. The omission of a hint may reflect its integration into earlier processing steps rather than its irrelevance to the final decision.

The authors further support this interpretation through Causal Mediation Analysis, following Pearl's framework for causal inference. Even when CoTs do not explicitly verbalize hints, the reasoning traces can still causally mediate the influence of those hints on the final prediction. This demonstrates that non-verbalized hints are not ignored by the model; rather, they are processed and transmitted through the CoT in ways that do not require explicit mention. The CoT functions as a genuine computational intermediate, not merely a confabulated narrative constructed after the fact.

Your Take: The Ontology of Explanation in Neural Networks

This research exposes a deeper conceptual tension in AI interpretability that extends beyond methodological quibbles. We are witnessing a category error in how we conceptualize explanation in neural systems. Current evaluation frameworks implicitly assume that faithfulness requires isomorphism between the explanation and the underlying computation; every causal factor must be explicitly represented. However, this standard is neither realistic for artificial systems nor demanded of human explanations.

Consider the architecture of reasoning in these models. We can conceptualize three distinct layers: the distributed parallel computation occurring across transformer layers, the working memory instantiated in the CoT tokens, and the final output prediction. Biasing Features effectively requires that information flow from layer one to layer three must be explicitly mirrored in layer two. But human cognition rarely meets this standard. When we solve mathematical problems or make decisions, our verbal explanations are selective, compressed accounts of cognitive processes that involve massive parallel processing, unconscious priming, and implicit associations.

The critical distinction lies between compression artifacts and confabulation. The former represents necessary selectivity in explanation generation; the latter represents misrepresentation of the actual reasoning process. Zaman and Srivastava's work suggests that current evaluations systematically mistake compression for confabulation. This has significant implications for AI safety and alignment research. If we dismiss CoT as an audit signal based on overly stringent criteria, we lose a valuable window into model cognition. Conversely, if we optimize models exclusively for hint verbalization, we risk creating systems that prioritize saying the right words over performing the right computations, a form of alignment theater that obscures rather than reveals true reasoning.

The Causal Mediation Analysis results are particularly significant for understanding how information propagates through these systems. That non-verbalized hints can still mediate predictions through the CoT suggests that the reasoning trace contains implicit markers or structural features that encode the hint's influence even without explicit mention. This aligns with recent work on latent representation interpretability, where model states encode information that may not be directly translatable into natural language tokens. Future interpretability research should investigate what aspects of hints are preserved in these traces; perhaps semantic features, confidence markers, or attentional patterns transmit the biasing information through the reasoning chain.

Conclusion

The question of whether Chain-of-Thought constitutes genuine explainability cannot be answered with single-metric evaluations. Zaman and Srivastava's work demonstrates that Biasing Features, while useful, provides an incomplete and potentially misleading picture of model faithfulness. By incorporating Causal Mediation Analysis, faithful@k metrics, and comparative evaluation across multiple faithfulness tests, we arrive at a more nuanced understanding. CoT reasoning appears to be neither perfectly transparent nor fundamentally deceptive, but rather a compressed, selective representation of computational processes that improves in fidelity when given adequate expression budget.

Moving forward, the interpretability community must adopt a more pluralistic approach to evaluation. No single metric can capture the multifaceted relationship between neural computation and natural language explanation. We need frameworks that distinguish between incompleteness and unfaithfulness, that account for the compression inherent in sequential explanation generation, and that recognize the causal role of reasoning traces even when they lack explicit verbalization. Only then can we accurately assess whether models are reasoning well, reasoning poorly, or merely reasoning differently than their explanations suggest.

The broader question remains open: as language models grow more capable and their reasoning becomes more sophisticated, will our methods for interpreting them scale accordingly? Or will we continue to demand that their explanations conform to simplistic, linear narratives that fail to capture the richness of distributed computation? The answer will determine whether we can effectively audit and align systems that increasingly shape critical decisions in science, medicine, and governance.