Decoding Structure, Not Just Category: What EEG-Based Visual Reconstruction Tells Us About the Brain

For more than a decade, the field of neural decoding has been chasing a deceptively simple goal: reconstruct what a person sees from recordings of their brain activity. The fMRI community has made the most visible progress here, producing reconstructions that, while imperfect, are often recognizable. EEG has lagged behind, hampered by its notoriously low signal-to-noise ratio and limited spatial resolution. A recent preprint, Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstruction (SGDM) by Lian, Cang, and colleagues at Tsinghua University, makes a contribution that I think is more interesting than the reconstruction images themselves. The paper argues, and provides evidence, that EEG carries explicit structural geometry, not merely categorical identity. That claim deserves careful unpacking.

The Limits of Category-Centric Decoding

Most EEG-based visual decoding pipelines follow a familiar recipe. An encoder maps EEG signals into an embedding space, typically aligned with CLIP via contrastive learning. The aligned features then condition a diffusion model, usually a variant of Stable Diffusion, to generate an image. This approach inherits the strengths and weaknesses of CLIP. It is excellent at semantic discrimination, so the reconstructed image tends to capture the category of the object the subject viewed. A dog stimulus yields a dog. A guitar yields a guitar.

The problem is that semantic category is a coarse target. As the authors note, three limitations recur across the literature:

• Over-reliance on categorical representations, which generalizes poorly to complex cognitive scenarios.
• Lack of explicit hierarchical structural information, meaning models capture color and contour but not local-to-global structural relationships.
• Difficulty aligning individual cognitive interpretations, leaving a semantic gap between generic neural signals and how a specific person parses an object.

The deeper issue is conceptual. Aligning EEG to CLIP embeddings ensures numerical similarity in a high-dimensional space, but the distributional mismatch between EEG and visual-semantic space is enormous. The result is unstable training, overfitting, and weak cross-subject generalization. You can match vectors without matching meaning. SGDM tries to break out of this trap by introducing a target that is neither purely categorical nor purely pixel-level: explicit structure.

How SGDM Introduces Structure

The architecture is a two-stage generative system with three coordinated components.

First, a structurally supervised variational autoencoder (VAE) learns a latent space that is explicitly organized around structural geometry rather than just appearance. This is the key departure. Instead of asking the latent space to encode "what kind of thing is this," the supervision pushes it to encode "how is this thing built from parts."

Second, a spatiotemporal EEG encoder maps raw EEG into a visual embedding space using contrastive learning. The spatiotemporal emphasis matters: EEG's value is its millisecond temporal resolution, and the encoder is designed to exploit the time course of neural responses, not just spatial topography across electrodes.

Third, and most importantly, structural information is injected into the diffusion model through ControlNet. ControlNet, originally introduced for conditioning image diffusion on edge maps, depth maps, or pose skeletons, is repurposed here to condition generation on EEG-derived structure. This is an elegant fit. ControlNet was built precisely to provide spatial-structural guidance to a diffusion backbone while preserving the generative prior. Using it to inject brain-decoded structure is a natural and, in retrospect, almost obvious move that the field had not fully exploited.

The evaluation design is what separates this paper from incremental work. The authors test on two contrasting datasets:

• THINGS, a large natural image dataset, representing the conventional setting where category and natural-image priors carry much of the signal.
• Kilogram, an abstract visual object dataset. Kilogram consists of tangram-like abstract shapes that lack obvious semantic category. You cannot decode a tangram by retrieving "dog" from CLIP, because there is no dog. To reconstruct it, you must decode its structure.

Testing on Kilogram is the methodological masterstroke. If a model reconstructs abstract shapes from EEG, it cannot be cheating via categorical retrieval. The authors report that SGDM outperforms existing methods on both datasets, with higher fidelity in low-level visual features and semantic representations. Crucially, the spatiotemporal analysis of EEG reveals hierarchical structural encoding patterns consistent with known neural dynamics of visual cognition.

Why the Abstract-Shape Result Is the Real Finding

I want to dwell on why the Kilogram result matters more than the headline reconstruction quality, a point I raised in a shorter note and want to develop properly here.

When a decoder reconstructs natural images, it is genuinely hard to know how much of the success comes from the brain signal and how much from the generative prior. Diffusion models trained on billions of natural images carry strong assumptions about what plausible images look like. A small nudge in the right semantic direction can produce a convincing reconstruction because the model fills in the rest. This is a well-known confound in fMRI reconstruction work as well. The image looks impressive, but the bits of information actually decoded from the brain may be few.

Abstract shapes strip away this crutch. There is no rich natural-image prior to lean on for a tangram. If the reconstruction preserves the spatial arrangement of parts, that structural information had to come from the EEG signal. So the Kilogram results function as a kind of ablation on the generative prior itself. They provide evidence that EEG encodes explicit geometric structure that can be decoded and used to constrain generation.

This connects to a broader principle from visual neuroscience. The hierarchical organization of the ventral stream, documented in work from Felleman and Van Essen through Yamins and DiCarlo, implies that the brain represents objects compositionally, building global structure from local features across processing stages. SGDM's spatiotemporal analysis showing hierarchical structural encoding across time windows is consistent with this picture. Early time windows may carry local feature information; later windows may carry integrated global structure. EEG, with its temporal resolution, is arguably better suited to capturing this temporal hierarchy than fMRI, which is what makes the modality choice interesting rather than merely a compromise for cost and accessibility.

My Take: Structure Precedes Semantics for BCI

The framing I find most compelling is the distinction between objective perception and subjective cognition, and the suggestion that ControlNet conditioning on EEG-derived structure can begin to separate them. The authors leverage human-annotated structural priors to bridge the gap between generic neural signals and individual interpretation. In a Kilogram-style task, two people can look at the same tangram and decompose it differently, seeing a rabbit versus a dancer. The structure they impose is subjective. If EEG can decode that imposed structure, you are reading interpretation, not just stimulus.

For brain-computer interfaces, this reorders priorities. Most BCI intention-decoding work targets discrete commands or category labels, which is a low degree-of-freedom channel. But intention is structural before it is semantic. When you imagine reaching for an object, arranging a layout, or sketching a shape, the structural plan precedes any verbal category. A decoder that recovers structural geometry from neural signals offers a much higher-dimensional control channel than one that classifies among a fixed label set. That is the practical prize hiding behind the reconstruction images.

Limitations and Open Questions

Several caveats temper my enthusiasm. The paper does not, in what I have seen, report cross-subject generalization in detail, and EEG decoding is notorious for failing to transfer across individuals. The reliance on human-annotated structural priors is also a double-edged sword: it improves supervision but raises the question of how much structure is genuinely decoded versus supplied by the annotation pipeline at training time. A clean test would quantify mutual information between EEG and reconstructed structure independent of annotations.

There is also the perennial generative-prior confound. Even with abstract shapes, ControlNet plus a diffusion backbone imposes priors. Rigorous evaluation should include identification metrics, asking whether a reconstruction can be matched to its true stimulus among distractors, rather than perceptual similarity alone.

Open questions I would prioritize:

• Does the temporal hierarchy of structural encoding map onto specific ERP components or oscillatory bands?
• Can the structure decoder generalize to imagined rather than perceived shapes, the true BCI use case?
• How does structural fidelity degrade as stimulus complexity increases?

SGDM does not resolve these, but by making structure an explicit and testable target, it gives the field a sharper question to ask than "how good does the picture look." That reframing, more than any single reconstruction, is the contribution worth remembering.