Causal Tokenization Meets MeanFlow: A Closer Look at CaTok

Visual tokenization sits at an awkward intersection of two design philosophies. Autoregressive language models thrive on causal sequences, where each token depends only on those before it, enabling clean next-token prediction. Images, by contrast, are inherently two-dimensional and non-sequential. Flattening a 2D grid of patches into a 1D sequence imposes an artificial ordering that rarely reflects any meaningful structure. The recent paper CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization by Chen, Wu, Qiu, and Jiang tackles this mismatch directly, proposing a 1D causal tokenizer whose token ordering encodes a coarse-to-fine progression. In doing so, it connects causal autoregressive modeling, progressive coding, and the recently popular MeanFlow generative objective.

This article unpacks how CaTok works, why its design choices matter, and where I think the approach succeeds and where open questions remain.

The Causality Problem in Visual Tokenizers

Existing visual tokenizers fall into two broad camps, each with a structural flaw.

The first camp flattens 2D patches into a sequence. Whether using raster order or some heuristic scan, the resulting tokens are non-causal in any semantically meaningful way. Token 50 is not a refinement of token 49; it simply happens to be a neighboring patch. This breaks the assumption that next-token prediction reflects a natural generative process.

The second camp involves diffusion autoencoders and methods that condition a decoder on a set of latent tokens. Here the issue is different. When the decoder sees all tokens simultaneously, there is no notion of ordering or causality at all. Attempts to impose ordering through nested dropout, where later tokens are randomly masked during training to encourage early tokens to carry more information, introduce a training imbalance: early tokens are seen far more often than late tokens, leading to uneven optimization and unstable representations.

CaTok's central insight is to bind token ordering to a temporal interval rather than to a dropout schedule. Tokens are selected over time intervals and tied to a MeanFlow objective. This reframes the question from "which tokens do I drop?" to "which interval of the generative trajectory does this token segment govern?" The result is a 1D representation where prefixes of the sequence reconstruct coarse structure and progressively longer prefixes refine detail.

Binding Tokens to MeanFlow

MeanFlow is a generative framework that learns an average velocity field over an interval, rather than the instantaneous velocity used in standard flow matching or rectified flow. The practical appeal is one-step sampling: because the model learns the integrated displacement across an interval, it can map noise to data in a single forward pass while retaining the option to take multiple steps for higher fidelity.

CaTok exploits this interval structure. The MeanFlow decoder is parameterized as a function of the noised latent, the interval endpoints, and a token segment corresponding to that interval. In the paper's notation, the decoder receives the latent, the start and end times of the interval, and the tokens indexed over that interval. By aligning token segments with intervals along the generative trajectory, the ordering of tokens acquires a concrete meaning: earlier intervals correspond to the coarse global transport from noise to a rough image, while later intervals handle fine refinement.

This is what produces the behavior shown in the paper's reconstruction figures. Truncating from 256 tokens down to 16 yields a graceful fine-to-coarse degradation rather than a catastrophic collapse. Different 16-token segments, when decoded in isolation, recover distinct visual concepts. The tokens have effectively learned a decomposition of the image into ordered components.

The authors report strong quantitative results on ImageNet reconstruction: 0.75 FID, 22.53 PSNR, and 0.674 SSIM, achieved with fewer training epochs than competing methods. The downstream autoregressive model built on these tokens reaches performance comparable to leading approaches. Importantly, the method supports both one-step generation (fast) and 25-step sampling (higher quality) from the same tokens.

REPA-A: Stabilizing Training Through Feature Alignment

A recurring difficulty in training tokenizers and diffusion-based decoders is slow or unstable convergence. CaTok introduces a regularization the authors call REPA-A, which aligns the encoder's intermediate features with representations from a Vision Foundation Model (VFM). This builds on the broader REPA line of work, which showed that aligning diffusion model internal features with pretrained self-supervised representations (such as DINOv2) accelerates training and improves sample quality.

The "A" variant here applies alignment in the tokenizer's encoder rather than the generative backbone. The intuition is that VFM features already encode a useful semantic hierarchy, and nudging the encoder toward this geometry gives the causal tokens a head start in organizing information from coarse semantics to fine detail. The reported efficiency gains, namely state-of-the-art reconstruction with fewer epochs, are consistent with this reasoning. Feature alignment regularization is becoming a standard tool, and CaTok's contribution is showing it transfers cleanly to the causal tokenization setting.

My Take: Learned Wavelets Without Fixed Bases

As I noted in my earlier short summary, the most striking conceptual parallel is to multiresolution wavelet decomposition. In a wavelet transform, an image is decomposed into a hierarchy of subbands: coarse approximation coefficients plus successive levels of detail. Truncating the high-frequency coefficients yields a smoothly degraded image, and different subbands isolate different frequency content. CaTok's tokens behave analogously. Early tokens carry global structure, later tokens refine, and distinct segments capture distinct concepts.

The crucial difference is that wavelets use fixed, hand-designed basis functions, whereas CaTok learns the ordering end-to-end and binds it to a MeanFlow objective. This is more flexible because the decomposition adapts to the data distribution rather than to a predetermined mathematical basis. It is also potentially more powerful, since the "subbands" can encode semantic rather than purely spectral structure. A learned coarse component might isolate object identity rather than low spatial frequencies.

This connection suggests a useful framing: CaTok is performing learned progressive coding with a generative decoder. Progressive coding has a long history in image compression (think progressive JPEG), and the field has recently rediscovered its value for token budgets, adaptive inference, and streaming generation. The fact that 0.75 FID is reachable with one-step sampling is the result I find most informative. It indicates that causality and progressive coding are not in tension with fast generation. One might have worried that enforcing an ordering would sacrifice the quality or speed advantages of MeanFlow, but the evidence here suggests otherwise.

A further inference: because token segments map to trajectory intervals, CaTok implicitly provides an adaptive compute mechanism. A downstream system could decode only the prefix it needs, spending more tokens and more sampling steps only when high fidelity is required. This is attractive for embodied or resource-constrained settings, which aligns with the authors' institutional focus on embodied AI.

Limitations and Open Questions

Several caveats deserve honest mention.

• Reconstruction is not generation. The headline 0.75 FID is a reconstruction metric. The autoregressive generation results are described as comparable to leading approaches rather than clearly superior, so the practical generative payoff of causal ordering remains to be fully demonstrated.
• Scaling behavior is unclear. Results are on ImageNet at a fixed token count. Whether the coarse-to-fine ordering holds at higher resolutions or with much longer sequences is unverified.
• Dependence on VFMs. REPA-A inherits whatever biases the underlying foundation model carries. The reliance on external pretrained features complicates clean ablation of where the gains originate.
• Interval design. How intervals are chosen and how sensitive the method is to this schedule is a design choice that may require tuning across datasets.

The broader open question is whether learned causal tokens can become the standard interface between vision and autoregressive language models, replacing both flattened patch sequences and non-causal latent sets. If they can, the payoff is a unified next-token framework spanning modalities. CaTok is a concrete step toward that goal, and its marriage of causal ordering with MeanFlow is, to my reading, the most thoughtful part of the design. I will be watching whether the learned-wavelet behavior survives at scale and whether the autoregressive generation quality eventually overtakes, rather than merely matches, non-causal baselines.