Where Identity Lives: Rethinking MLLM Conditioning for Subject-Driven Generation

Subject-driven image generation sits at an awkward intersection. On one side, we want the model to faithfully reproduce a specific subject, the user's particular dog, backpack, or stuffed animal, down to its fine visual details. On the other, we want the model to follow textual instructions that may demand substantial deviation from the reference image: place the backpack in the snow, on pink fabric, surrounded by sunflowers. These two goals pull in opposite directions, and most systems end up sacrificing one for the other. The recent paper Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation (Zheng et al.) offers a careful diagnosis of why this happens and a set of architectural choices to address it.

The central claim, which I summarized in a recent post, is deceptively simple. The standard practice of conditioning diffusion models on the final-layer features of a multimodal large language model (MLLM) is the wrong default. Final layers are optimized for semantic abstraction, and in the process they discard exactly the fine-grained visual cues that identity preservation depends on. The fix is to aggregate across multiple layers and then balance semantic versus detail conditioning during the denoising trajectory. Below I want to unpack why this is a meaningful observation, what the paper actually does, and where I think the limits of the approach lie.

The Two Lineages and Their Failure Modes

It helps to understand the two research lineages the paper is trying to unify.

The first lineage is VAE-based conditioning. Methods in this family, including the more recent zero-shot systems, encode the reference image into VAE latent tokens and feed those into the diffusion process. Because VAE latents preserve high-frequency spatial detail, this gives strong identity fidelity. The problem is that text and image are encoded separately, so the model has weak cross-modal reasoning. The most visible symptom is the copy-paste artifact: the subject gets pasted into the scene with little regard for the prompt's spatial or contextual demands, or with identity drift on prompts that require genuine recomposition.

The second lineage is MLLM-driven conditioning. Here the idea is to let an MLLM jointly parse interleaved text and image instructions, then condition a diffusion decoder on the MLLM's output. Systems like DreamEngine, Qwen-Image, and EasyRef follow this template. The benefit is much better instruction following and flexible prompt interpretation, because the MLLM does real multimodal reasoning. The cost is identity preservation. As the paper notes, these systems typically rely on the final-layer features alone (Qwen-Image, EasyRef), or combine ViT features with final-layer outputs via scalar mixing (DreamEngine). Either way, the fine-grained visual information needed for identity gets squeezed out.

The paper's contribution is to take the reasoning strength of the MLLM lineage and recover the identity strength of the VAE lineage, without letting them cancel each other out.

Dual Layer Aggregation and Staged Denoising

Two mechanisms carry most of the weight here.

The first is the Dual Layer Aggregation (DLA) module. Rather than conditioning on a single layer, DLA performs layerwise attention pooling across all transformer layers of the MLLM. Crucially, it treats text and image tokens separately. This separation is the part I find most principled. Text tokens and image tokens in an MLLM have genuinely different feature structures and statistics; naively fusing them or collapsing them into a single representation throws away information about which modality contributes what. By aggregating text and visual embeddings along independent paths, DLA can pull semantic context from where it lives (later layers) and fine visual detail from where it lives (earlier and middle layers).

This connects to a broader body of interpretability work on transformers, where the standard finding is that representations evolve from surface and local features in early layers toward abstract, task-relevant features in later layers. The same gradient appears to hold inside MLLMs for visual tokens. The paper's analysis of early, middle, and late layer groups makes this explicit, and it is consistent with what we have seen in vision transformers more generally: top layers are great for classification-style semantics and poor for reconstruction-style detail. Conditioning a generative model solely on the top is, in retrospect, an obvious mismatch.

The second mechanism addresses a subtler problem. Once you add VAE-based identity conditioning on top of MLLM conditioning, the two signals overlap. Both encode visual information about the subject, and naively combining them produces embedding conflicts. The paper's answer is twofold:

• A two-stage training strategy that first establishes multimodal conditioning from the MLLM, then introduces the high-frequency identity details from the VAE.
• A multi-stage denoising strategy at inference. The diffusion model first denoises under MLLM guidance to lay down global semantics, then refines jointly with both modalities, and finally focuses on VAE-conditioned fine details.

This denoising schedule is well-aligned with what we know about diffusion dynamics. Early denoising steps determine global structure and layout, while late steps fill in texture and detail. Routing semantic MLLM guidance to the early steps and VAE detail to the late steps is not arbitrary; it matches the natural information schedule of the reverse diffusion process. This is a clean example of architecture design that respects the inductive structure of the underlying generative model rather than fighting it.

My Take

What I appreciate about this work is that it reframes a conditioning problem as an information allocation problem. The question is not simply "which features do we use" but "where does each kind of information live, and at what stage of generation does it matter." Both DLA and the staged denoising answer different facets of that single question, one across MLLM depth, the other across diffusion time.

A few inferences and concerns worth stating honestly:

• The win is likely conditional on subject complexity. For simple, rigid objects the VAE branch alone may already be adequate, and the MLLM machinery mostly helps on prompts that require recomposition or unusual scene context. I would expect the gains to concentrate exactly where prior methods produce copy-paste artifacts. The qualitative examples (backpack in snow, on pink fabric, among sunflowers) seem chosen to highlight this.

• Attention pooling over all layers is not free. Aggregating across every transformer layer multiplies the conditioning footprint and adds parameters and memory. The paper frames efficiency as orthogonal, but in practice the layerwise approach competes with single-layer methods on compute, and a fair comparison should account for that.

• The evaluation leans on human preference. Identity preservation and instruction following are notoriously hard to measure with automated metrics, so human preference is reasonable. But it makes the results harder to reproduce and harder to compare across papers. I would want to see DINO or CLIP-I identity similarity alongside CLIP-T prompt alignment to triangulate the human judgments.

• The conflict between branches is managed, not eliminated. Two overlapping visual signals will always carry redundant and occasionally contradictory information. The staged schedule mitigates this, but I suspect there is residual tension on prompts where the desired output is genuinely far from the reference.

Closing Thoughts

The broader lesson generalizes beyond subject-driven generation. As we increasingly use MLLMs as conditioning engines for downstream generative models, the habit of grabbing the final hidden state and moving on deserves scrutiny. Final-layer representations are optimized for whatever objective trained the MLLM, which is rarely image reconstruction. The detail you need for generation often lives in the intermediate representations, and recovering it is a question of designing the right readout, not just the right backbone.

The open questions I find most interesting are about learned routing. Could the layer aggregation and the denoising schedule be made adaptive per prompt, so that recomposition-heavy instructions draw more from semantic layers while faithful-reproduction instructions draw more from detail layers? And can we characterize, layer by layer, exactly what visual information an MLLM retains versus discards? Answering that would turn the empirical findings here into a predictive theory of how to read information out of multimodal models for generation.