Beyond Activation Patterns: Causal Decomposition of Neural Network Computations with CODEC

Introduction

The dominant paradigm in neural network interpretability has long been an exercise in pattern recognition. We visualize what makes a neuron fire, cluster activation vectors to find latent concepts, or use sparse autoencoders to extract monosemantic features from hidden representations. Yet these approaches share a fundamental limitation. They tell us what networks see, not what they do. The paper Causal Interpretation of Neural Network Computations with Contribution Decomposition, presented at ICLR 2026 by Melander, Alaoui, Liu, and colleagues, proposes a decisive shift in framework. Rather than analyzing the receptive field of network components, what inputs cause a neuron to activate, the authors examine the projective field: how each component causally influences downstream computation and final output.

This distinction is not merely semantic. In biological neural circuits, a retinal ganglion cell might respond to light onset in a specific spatial location, but its contribution to behavior depends on which brain areas it projects to and whether those projections drive or suppress specific actions. Similarly, in deep convolutional networks, a feature detector for vertical edges might activate on street signs in one image and window frames in another, yet only the former contributes to classifying the image as a stop sign. The authors introduce CODEC (Contribution Decomposition) to disentangle this causal structure, using sparse autoencoders to decompose network behavior into interpretable motifs of hidden neuron contributions. Their findings challenge several assumptions about how hierarchical processing operates, particularly the unexpected discovery that positive and negative influences on network outputs progressively decorrelate across layers.

From Correlation to Causation: The Currency of Contribution

Most interpretability methods treat hidden activations as the primary object of study. Sparse autoencoders trained on layer activations identify directions in representation space that correspond to human interpretable concepts. Feature visualization reveals preferred stimuli for individual channels. However, as the authors note, activation patterns reflect only the receptive field stage of computation. A high activation indicates presence of a feature, but says nothing about whether that feature is being used, ignored, or actively suppressed in the service of the final output.

CODEC addresses this gap by computing the contribution of each hidden unit, defined as the composition of receptive and projective fields. Technically, the authors extend gradient based attribution methods to compute how each hidden unit influences scalar targets of network output. These targets include not just individual class logits, but also aggregate measures such as the entropy of the output distribution and the sum of top k logits. This produces a contribution map for each unit, quantifying its positive or negative causal influence on the specific computation being analyzed.

The crucial innovation follows. Rather than applying sparse coding to activation matrices, CODEC applies sparse autoencoders to the contribution matrices themselves. This decomposes network behavior into sparse motifs, or contribution modes, that represent coordinated patterns of causal influence across populations of neurons. The difference is subtle but profound. Where activation based sparse coding asks "what features are present in this layer," CODEC asks "what combinations of neurons work together to drive or suppress specific outputs."

The Architecture of Influence: Sparsity, Dimensionality, and Decorrelation

Applying CODEC to ResNet 50 on standard image classification benchmarks reveals three empirical patterns that reshape our understanding of how information flows through deep networks.

First, contributions grow progressively sparser across layers. Early layers show dense contribution patterns where many units weakly influence the output. By late layers, influence concentrates in sparse subsets of neurons with strong causal effects. This suggests a gradual refinement from distributed, redundant coding to sparse, selective routing of information.

Second, the effective dimensionality of contribution modes increases with depth. While early layers might encode simple, low dimensional motifs such as edge combinations, deeper layers express higher dimensional, combinatorial modes that integrate multiple feature types. This aligns with theoretical predictions about hierarchical processing but provides direct causal evidence rather than correlational observations.

The third finding is the most conceptually significant and, as noted in discussions of this work, deserves greater attention. CODEC reveals that positive and negative contributions to network outputs progressively decorrelate across layers. In early layers, the same neurons often exert both excitatory and inhibitory influences on different aspects of the output. In deeper layers, these functions segregate. Populations emerge that specialize in driving specific classes while other populations specialize in suppressing alternatives.

This decorrelation has profound implications. It suggests that deep networks learn to separate excitatory and inhibitory pathways, not at the level of individual synapses as in biological neurons, but at the level of population codes. The functional architecture resembles a switchboard where approval and veto rights are delegated to distinct subnetworks. This organizational principle likely facilitates stable credit assignment during training, allowing the network to adjust "pro" and "con" evidence independently without destructive interference.

Original Insights and Limitations

The CODEC framework represents a maturation of mechanistic interpretability from descriptive to causal analysis. By establishing contribution modes as the fundamental unit of study, it bridges the gap between circuit tracing and population level statistics. However, several limitations and open questions warrant consideration.

The method relies on gradient based attribution, which carries inherent constraints. Gradient saturation, path dependence in non convex loss landscapes, and the shapley value problem of fairly attributing marginal contributions all apply here. The authors compute contributions to scalar targets, which requires aggregating across spatial dimensions in convolutional layers. While this enables the sparse decomposition, it may obscure fine grained spatial structure in how contributions are organized.

More fundamentally, the decorrelation of positive and negative contributions raises questions about the geometry of decision boundaries in deep networks. If late layer networks segregate into "advocate" and "critic" populations, this suggests a form of functional specialization that resembles ensemble methods or mixture of experts architectures, but emergent rather than explicitly engineered. One wonders whether this segregation is a general property of deep supervised learning, or whether it emerges specifically in architectures with skip connections like ResNet, where gradient pathways might encourage such factorization.

The application to retinal models is particularly promising. Biological neural networks have long served as inspiration for artificial architectures, but CODEC enables reverse translation. By revealing combinatorial actions of model interneurons and identifying sources of dynamic receptive fields, the method generates testable hypotheses about biological circuits. The finding that artificial retinal models exhibit sparse contribution modes suggests that biological retinas might similarly employ sparse, causal population codes rather than dense distributed representations.

Conclusion

The shift from analyzing activations to analyzing contributions marks a necessary evolution in how we understand neural computation. The CODEC paper demonstrates that what a network detects and what it does with that detection are distinct phenomena, and that the latter is where causal structure resides. The discovery that contributions grow sparser, higher dimensional, and functionally segregated by sign across layers provides empirical grounding for theories of hierarchical processing that previously relied on correlational evidence.

Looking forward, several questions remain. Does the decorrelation of positive and negative contributions hold in transformer architectures, where attention mechanisms might recombine excitatory and inhibitory signals in more complex ways? Can contribution modes serve as intervention targets for model editing, allowing precise manipulation of network behavior without full retraining? And how do contribution structures evolve during training, particularly during the transition from memorization to generalization?

What remains clear is that contribution analysis offers a richer vocabulary for describing network function than activation analysis alone. As we seek to build safe, interpretable, and modifiable AI systems, understanding the causal currency of computation, not merely its correlational signatures, will be essential. CODEC provides both the methodological tools and the conceptual framework to pursue this understanding across biological and artificial systems alike.