Rethinking Vector Field Learning for Generative Segmentation: An Optimization Dynamics Perspective

Introduction

The apparent universality of diffusion models has prompted researchers to apply them to virtually every computer vision task imaginable. Yet while these models excel at synthesizing continuous visual data, their adaptation to discrete perception tasks remains problematic. Semantic segmentation, which assigns categorical labels to pixels, presents a particularly stark challenge. The continuous, stochastic nature of diffusion processes seems fundamentally at odds with the deterministic, discrete requirements of pixel classification.

In their paper "Rethinking Vector Field Learning for Generative Segmentation," Wang et al. from Peking University and Baidu argue that the issue lies not with model capacity or architectural choices, but with the optimization dynamics of the flow matching objective itself. Their analysis reveals two critical pathologies that plague generative segmentation: gradient vanishing near semantic centroids and trajectory traversing through competing class regions. This work offers both a diagnostic framework for understanding these failures and an elegant corrective mechanism that reshapes the learned vector field without dismantling the core diffusion training paradigm.

The Pathology: Gradient Vanishing and Trajectory Traversing

To understand why standard flow matching falters in segmentation, we must examine the geometry of the target space. In image generation, diffusion models learn to map noise to complex, high dimensional data manifolds where each sample is unique. Semantic segmentation, however, requires mapping pixels to a finite set of class centroids residing in a bounded, low dimensional space. This structural difference has profound implications for optimization dynamics.

Under the standard flow matching formulation with x prediction, the loss induces gradients proportional to the vector difference between the current prediction and the target centroid. Specifically, the gradient scales as ∇_x L ∝ (x - µ), where µ represents the semantic centroid. As the denoising trajectory approaches the target, this difference shrinks, causing gradients to vanish exponentially. The optimization stalls precisely when precision matters most, resulting in slow convergence and uncertain final predictions.

Compounding this issue is the phenomenon the authors term trajectory traversing. Standard flow matching provides only attractive forces toward the ground truth centroid, with no mechanism for repulsion from competing classes. In a multi class scenario, denoising paths must navigate through a field of nearby centroids. Without explicit repulsive gradients, trajectories wander through proximal neighborhoods of irrelevant classes before converging. This creates ambiguous intermediate states and increases the risk of misclassification, particularly for visually similar categories.

The authors visualize this through the lens of decision