Beyond Directions: Do Sparse Autoencoders Capture Concept Manifolds?

The mechanistic interpretability community has long operated under the assumption that neural networks represent abstract concepts as directions in activation space. This Linear Representation Hypothesis (LRH) posits that concepts correspond to isolated vectors, and that the geometry of neural representations can be understood as an additive mixture of these independent components. Sparse Autoencoders (SAEs), which have emerged as the primary tool for unsupervised feature discovery in large language models, are built upon this assumption. By enforcing sparsity constraints through L1 regularization, SAEs attempt to decompose neural activations into interpretable dictionaries of independent features.

However, a growing body of evidence suggests that many concepts are not isolated directions but rather organized along low-dimensional manifolds encoding continuous geometric relationships. In their recent work, "Do Sparse Autoencoders Capture Concept Manifolds?", Bhalla et al. rigorously examine whether SAEs can capture these more complex geometric structures. Their findings reveal fundamental limitations in how current SAE architectures handle continuous concepts, and they propose that the field must move beyond individual directions toward geometric objects as the basic units of interpretability.

The Geometry of Representation: From Vectors to Manifolds

The LRH assumes that concepts like "age" or "temperature" can be represented as single directions in activation space, packed efficiently as Grassmannian frames. Yet empirical observations tell a different story. When examining activations from Llama 3.1-8B at layer 19, Bhalla et al. find that continuous concepts such as age (ranging from young to elderly), temperature (freezing to burning), days of the week, and color hue exhibit smooth, curved geometric structure rather than isolated linear directions. Principal Component Analysis (PCA) projections of these activations reveal one-dimensional curves that preserve semantic relationships; points corresponding to similar ages cluster together along continuous trajectories.

This manifold structure has causal efficacy. The authors demonstrate that steering interventions between concept centroids, such as moving from Wednesday to Thursday representations, produce smooth changes in token probabilities for concept-dependent outputs. This confirms that the geometric continuity in activation space directly influences model behavior, validating that these manifolds are not merely artifacts but functional components of the network's computation.

Given this geometry, the authors ask a critical question: how can SAEs, which assume sparse linear superposition of features, represent these continuous structures? They identify two fundamentally different mechanisms through which SAEs might capture manifolds. Global capture occurs when an SAE allocates a compact group of dictionary atoms (features) whose linear span contains the entire manifold. In this regime, the manifold is represented by the combined activation pattern of a specific subset of features working in concert. Local capture, conversely, occurs when the manifold is distributed across features that each selectively tile restricted regions of the underlying geometry, with different atoms activating for different values along the continuous spectrum.

The Dilution Problem: When SAEs Fragment Continuous Structure

Empirically, Bhalla et al. find that SAEs do not cleanly adopt either the global or local strategy. Instead, they observe a dilution regime where SAEs suboptimally mix both approaches. Rather than allocating a coherent subspace to represent a continuous concept globally, or cleanly partitioning the manifold into interpretable local tiles, SAEs fragment the representation across many atoms. This fragmentation explains why manifold structure is rarely visible when examining individual SAE features in isolation.

The technical details of this finding are illuminating. When training SAEs with standard L1 regularization on activations containing known manifolds, the reconstruction objective forces the model to preserve the geometric relationships necessary for accurate output. However, the sparsity constraint works against this goal for continuous concepts. A perfectly sparse representation would require activating only a few neurons for any given input, but representing a point on a continuous manifold precisely requires coordinated activation patterns that encode position along the curve. The tension between these objectives results in dilution: many atoms partially participate in representing the manifold, each capturing small, overlapping regions without clear semantic coherence.

This has significant implications for how we interpret SAE features. Current automated interpretability pipelines often search for individual atoms that correlate with specific concepts. Under the dilution regime, such methods fail because no single atom cleanly represents "middle age" or "warm temperature." Instead, these concepts exist only as emergent properties of coordinated groups of atoms. The authors therefore advocate for post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions.

Rethinking the Inductive Biases of Interpretability

The findings of Bhalla et al. suggest a deeper issue with current interpretability methodologies. The field has prioritized sparsity as the primary inductive bias for feature extraction, assuming that interpretability requires decomposing neural activity into minimally overlapping components. However, for continuous concepts that inherently possess metric structure, sparsity may be the wrong bias entirely.

Consider the superposition hypothesis, which posits that neural networks represent more features than they have dimensions by compressing them into overlapping subspaces. For binary or categorical features, sparse coding works well. But for continuous variables like temperature or spatial position, the representation requires preserving relative distances and smooth transitions. An SAE forcing sparsity upon such structures must either use many atoms to approximate the continuum (dilution) or lose the fine-grained geometric relationships entirely.

This suggests that future representation learning methods for interpretability should incorporate geometric awareness. Rather than treating all concepts as discrete directions to be discovered, we might employ hierarchical architectures that explicitly model continuous dimensions, or utilize non-linear decoders that can map curved manifolds without forcing them into linear subspaces. Vector quantization methods or kernel-based approaches might better respect the topology of concept manifolds than current L1-regularized linear autoencoders.

Limitations and Open Questions

The study focuses primarily on one-dimensional manifolds embedded in the activation spaces of transformer models. Higher-dimensional manifolds, such as those encoding the joint variation of multiple continuous attributes, present additional challenges. The authors note that their definition of "manifold" remains partly a working assumption; whether real neural representations satisfy strict differential-geometric definitions remains unresolved.

Furthermore, the dilution phenomenon raises practical questions for current interpretability efforts. If individual SAE features are not interpretable for continuous concepts, but groups are, how do we efficiently discover these groups without supervision? The paper suggests unsupervised clustering of atoms, but this adds another layer of complexity to the interpretation pipeline. There is also the question of whether dilution is merely an inconvenience for human interpreters or if it indicates that SAEs are failing to capture the true computational structure used by the network during inference.

Conclusion

The work of Bhalla et al. marks an important correction to the trajectory of mechanistic interpretability. By demonstrating that Sparse Autoencoders capture continuous concept manifolds through a fragmented, diluted mixture of global and local strategies, they reveal the limitations of assuming that interpretability means finding isolated linear directions. The geometric richness of neural representations, with their curved manifolds and continuous transitions, demands tools that respect this structure.

As we develop the next generation of interpretability methods, we must consider whether sparsity remains the appropriate goal. Perhaps the basic unit of interpretability is not the individual feature but the geometric object; not the atom but the manifold. Moving forward, the field requires architectures that can explicitly model continuous variation, preserving the metric relationships that give meaning to distances in activation space. Only then will we truly understand how neural networks represent the continuous, graded nature of the world.