Preference-Conditioned Diffusion: Navigating the Pareto Front Without Retraining

Reinforcement learning post training has become the dominant paradigm for aligning diffusion models with human preferences. Yet the standard approach rests on a convenient fiction: that human judgment can be reduced to a single scalar reward. In practice, we rarely want just one thing. An image editing model must balance prompt adherence against source fidelity. A text to image system must navigate aesthetic quality versus textual alignment. These objectives exist in inherent tension, and the optimal balance varies by user, context, and creative intent.

Current methods address this multiplicity through early scalarization, collapsing competing rewards into a fixed weighted sum before training begins. This approach, while computationally convenient, commits the model to a static compromise. Once trained, the model cannot adapt to shifting preferences without expensive retraining or maintaining multiple checkpoints.

The paper ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control by Golan et al. offers a fundamental correction to this paradigm. Rather than baking the trade off into the model weights, they condition the model on a preference vector that specifies the desired balance at inference time. The result is a single model that approximates the entire Pareto front, allowing continuous navigation between competing objectives without additional training cost.

The Mechanics of Late Scalarization

The technical innovation of ParetoSlider lies in its treatment of the preference vector ω. During training, the model receives not just the input noise and conditioning, but also a vector specifying the relative importance of each reward. This conditioning is integrated into the diffusion transformer architecture, allowing the model to modulate its behavior based on the desired trade off.

However, simply aggregating rewards weighted by ω proves insufficient. Raw reward functions often operate on different scales and variances. Without careful normalization, high magnitude rewards dominate the gradient updates, effectively hijacking the optimization process and rendering the preference conditioning irrelevant. The authors address this through late scalarization: computing advantage estimates independently for each reward, normalizing them to comparable scales, and only then applying the preference weights ω for the final aggregation.

This approach contrasts sharply with early scalarization, where rewards are combined before gradient computation. By preserving the independence of reward signals until the final aggregation step, ParetoSlider ensures that the optimization landscape reflects the user specified preferences rather than the arbitrary scaling of underlying reward functions.

The framework builds upon DiffusionNFT, a flow matching RL fine tuning method, adapting it for multi objective optimization. For each training step, the model computes NFT losses for each reward independently, then aggregates them based on the sampled preference vector. This training procedure ensures that the model learns not a single policy, but a manifold of optimal policies parameterized by ω.

The conditioning mechanism itself warrants attention. The authors explore various architectural strategies for injecting ω into the diffusion transformer, finding that the method of integration significantly impacts the smoothness of transitions across the Pareto front. Simple concatenation or additive schemes prove inferior to carefully designed injection schemes that preserve the semantic structure of the preference vector throughout the network layers.

Empirical Validation Across Modalities

The authors evaluate their approach across three distinct generative domains, demonstrating remarkable consistency in performance gains. Using Stable Diffusion 3.5 for text to image generation, FluxKontext for instruction based image editing, and LTX 2 for text to video synthesis, they show that a single preference conditioned model matches or exceeds the performance of multiple separately trained baselines.

In the editing domain, the tension between source preservation and prompt adherence presents a particularly acute test. The results indicate that ParetoSlider not only achieves comparable performance to specialist models at specific trade off points, but enables smooth interpolation between them. This continuous control represents a qualitative shift from previous approaches that required discrete model switching or expensive interpolation between checkpoints.

The comparison with model interpolation baselines highlights the efficiency gains of the approach. Rather than training and storing N separate models for N different trade off points, ParetoSlider maintains a single set of parameters. This reduces storage overhead from linear to constant with respect to the granularity of control, while eliminating the inference time computational cost of switching between or blending multiple models.

Notably, the method demonstrates robustness across different backbone architectures. Whether applied to the rectified flow formulation of SD3.5, the context aware editing capabilities of FluxKontext, or the spatiotemporal dynamics of LTX 2, the preference conditioning achieves consistent results. This architectural agnosticism suggests that the principle of late scalarization and preference conditioning generalizes across the broader family of flow matching and diffusion models.

Your Take

What strikes me most about ParetoSlider is not merely the technical elegance, but the conceptual reorientation it represents. The field has spent years optimizing for single point solutions, treating the tension between objectives as something to be resolved before deployment. This work recognizes that the tension itself is information rich, and that the ability to navigate it dynamically constitutes a primary capability rather than a secondary feature.

From a product perspective, this changes how we should think about generative interfaces. Rather than offering users a fixed slider between "creativity" and "coherence", or maintaining separate models for "conservative" versus "aggressive" edits, we can offer continuous control spaces that adapt to individual creative workflows. The model becomes not a tool with fixed characteristics, but an instrument responsive to the performer's immediate intent.

There are limitations worth acknowledging. The method assumes that the Pareto front is sufficiently smooth to be approximated by a single network with continuous conditioning. For highly discontinuous trade off surfaces, or objectives that fundamentally require different architectural solutions, the approach may struggle. Additionally, the computational cost of training scales with the number of reward functions, and the paper does not explore cases with more than two objectives in depth. Whether the method maintains its efficacy for high dimensional preference vectors remains an open question.

Furthermore, the reliance on explicit reward functions raises questions about reward hacking and specification gaming. While late scalarization prevents one reward from dominating numerically, it does not address deeper misalignments between proxy rewards and human intent. As we expand the dimensionality of controllable rewards, we may simply multiply the opportunities for unintended behaviors. The paper assumes access to well defined reward functions, yet in practice, specifying rewards that capture human aesthetic judgment remains an unsolved problem.

Conclusion

ParetoSlider establishes a new baseline for multi objective alignment in generative models. By conditioning on preference vectors and employing late scalarization, it achieves what early scalarization promised but could not deliver: true inference time control over the reward trade off surface.

The work suggests a