Order Matters: How Flexible Sequencing Unlocks Autoregressive Models for Visual Classification

Introduction

The resurgence of generative modeling has produced remarkable systems capable of synthesizing photorealistic imagery, yet the discriminative potential of these models remains comparatively underexplored. While the research community has focused intensely on diffusion models for generative classification tasks, autoregressive approaches have languished in relative obscurity, dismissed perhaps due to their historical reliance on rigid, raster scan generation patterns. The recent work Revisiting Autoregressive Models for Generative Image Classification by Sudakov, Babenko, and Baranchuk from Yandex Research challenges this neglect, demonstrating that AR models can not only match but exceed the performance of diffusion based classifiers while operating with significantly greater computational efficiency.

At the heart of their investigation lies a subtle but critical observation: the fixed token ordering imposed by traditional AR architectures creates a restrictive inductive bias that fundamentally limits their capacity for image understanding. By leveraging recent advances in any order autoregressive modeling, the authors unlock a pathway to robust visual classification that challenges the current dominance of diffusion based approaches.

The Tyranny of Raster Order

Traditional autoregressive image models, from early vector quantized variational autoencoder (VQ VAE) based systems to contemporary next token prediction architectures, typically enforce a raster scan generation order. Images are processed sequentially from left to right and top to bottom, mimicking the linear structure of natural language. This architectural choice, while facilitating straightforward implementation, imposes a sequential bias ill suited to the fundamentally parallel and hierarchical nature of visual information.

Sudakov et al. identify this fixed ordering as a primary limitation for generative classification. Their analysis reveals that single order predictions rely excessively on partial discriminative cues available early in the sequence, rendering the model vulnerable to ambiguous local features. Figure 1 in their paper illustrates this phenomenon compellingly: identical images receive different classifications depending on whether the model processes tokens in a standard raster pattern, a spiral, or other permutations. Some orders correctly identify the subject while others fixate on distracting background elements.

This observation suggests that diffusion models, which generate images through a coarse to fine, noise conditioned process interpretable as spectral autoregression, benefit from an implicit ordering that better reflects the hierarchical structure of natural scenes. The spectral approach progresses from low to high frequency components, capturing global structure before local details. In contrast, spatial raster ordering forces the model to commit to early visual cues before accessing the full context of the image.

Order Marginalization and Computational Efficiency

To address the limitations of fixed ordering, the authors employ Rand