The Compression Gap: How Discrete Action Tokenization Limits Scaling in Visuomotor Models

A foundational assumption in Vision-Language-Action (VLA) research holds that upgrading the vision encoder should improve downstream manipulation performance. This principle appears robust in vision-language modeling, where moving from task-agnostic convolutional networks to semantically rich representations like SigLIP yields consistent gains across benchmarks. However, a recent study by Takuya Shiba titled The Compression Gap: Why Discrete Tokenization Limits Vision-Language-Action Model Scaling demonstrates that this assumption fails catastrophically when actions are represented as discrete tokens. The work reveals that the location of information bottlenecks within the visuomotor pipeline, not merely the quality of individual components, governs scaling behavior in Physical AI.

The Information Bottleneck Mechanism

The core insight of Shiba's analysis rests on an information-theoretic principle regarding bottleneck location in visuomotor architectures. In any pipeline mapping observations to actions, scaling behavior is constrained by the tightest information bottleneck; improvements upstream of this constraint cannot propagate downstream, regardless of their intrinsic quality.

The paper contrasts two representative pathways. In the continuous pathway, exemplified by Diffusion Policy, the vision encoder itself constitutes the binding bottleneck. The mutual information I(O; Z) between observation O and latent representation Z is limited by the encoder's capacity. Upgrading from ResNet 18 to SigLIP directly increases this mutual information, widening the information pathway to action prediction and yielding end to end performance gains.

In the discrete pathway, represented by Ordered Action Tokenization (OAT), the architecture introduces an additional compression stage. A learned tokenizer T compresses continuous action chunks into discrete tokens via Finite Scalar Quantization (FSQ), specifically using quantization levels [8, 5, 5, 5] to produce a codebook of size 1000. This imposes a hard information ceiling: I(Z; T) ≤ 80 bits. Regardless of how rich the upstream visual representation Z becomes, all action information must pass through this fixed capacity codebook. The codebook, not the vision encoder, becomes the binding constraint. Consequently, encoder upgrades are blocked at the quantization stage, unable to influence downstream action quality.

This structural asymmetry explains why two policies using identical observation pipelines can exhibit radically different scaling properties. The continuous pathway allows ungraded information flow from perception to action, while the discrete pathway truncates information through a lossy compression stage that is invariant to encoder improvements.

Empirical Evidence on LIBERO

Shiba validates this bottleneck hypothesis through three lines of experimental evidence on the LIBERO manipulation benchmark, each designed to isolate the interaction between encoder quality and action representation.

First, a factorial experiment compares encoder upgrades across model scales (Medium and Large). When upgrading from ResNet 18 to SigLIP, Diffusion Policy improves by over 21 percentage points in success rate. OAT, under identical conditions, shows substantially attenuated gains that remain inconsistent across scales. This interaction effect provides initial evidence that encoder sensitivity depends on action representation modality.

Second, an encoder quality gradient experiment tests four vision encoders of varying capability: ResNet 18, DINOv2, SigLIP, and SigLIP 2. Diffusion Policy exhibits monotonic improvement that tracks encoder quality; better encoders reliably yield better manipulation. OAT's performance curve remains essentially flat across the same gradient, confirming that once encoder quality exceeds a modest threshold, further improvements provide no marginal benefit to the discrete tokenization approach.

Third, and most compellingly, a codebook size ablation provides causal evidence for the bottleneck hypothesis. By increasing the capacity of OAT's FSQ codebook, the authors partially recover sensitivity to encoder quality. When the artificial 80 bit constraint is relaxed, OAT begins to benefit from SigLIP upgrades, demonstrating that the codebook capacity was indeed the limiting factor preventing information propagation.

These results collectively establish that discrete tokenization creates a compression gap; architectural choices regarding action representation determine whether scaling investments in perception translate to improved physical capabilities.

Broader Implications for Physical AI

The compression gap phenomenon extends beyond the specific comparison of OAT and Diffusion Policy. It illuminates a fundamental tension in current VLA design. Discrete tokenization has become popular because it enables the use of autoregressive transformer architectures, compresses action sequences into shorter token horizons (H_l << H_a), and facilitates cross-modal training with language. However, Shiba's findings suggest these advantages come at an informational cost that may impose hard ceilings on performance.

The 80 bit constraint in OAT is particularly striking when contextualized against the information content of visual scenes. Modern vision encoders like SigLIP extract thousands of bits of semantic and geometric information from images, yet OAT's action tokenizer can utilize at most 80 bits to determine motor outputs. This mismatch creates a severe representational bottleneck that no amount of encoder scaling can overcome. The situation parallels findings by Tong et al. (2026), who observed that VAE based visual representations saturate with model scale while semantic encoders continue improving. The convergence suggests that lossy compression stages create structural barriers to scaling across both perception and action domains.

For the field of robotic learning, this implies that current scaling laws may be artificially constrained by tokenization choices rather than fundamental limits in model capacity or data volume. Researchers investing in larger vision encoders or more diverse pretraining datasets may see diminishing returns if their action representations cannot accommodate the resulting information richness. The bitter lesson here concerns not just computation, but information flow; we must identify where in the pipeline information is actually lost before allocating resources to scale components upstream of those bottlenecks.

Limitations and Open Questions

The study focuses specifically on the LIBERO benchmark, which involves tabletop manipulation with relatively short action horizons. Whether the compression gap persists in high dimensional action spaces, mobile manipulation, or high frequency control remains an open empirical question. Additionally, OAT represents one specific instantiation of discrete action tokenization using FSQ; other quantization methods or learned codebook approaches might exhibit different capacity characteristics. Diffusion Policy, while avoiding the discrete bottleneck, requires multiple inference steps for denoising, raising deployment considerations that OAT's single forward pass avoids.

Conclusion

Shiba's analysis of the compression gap provides a necessary corrective to the assumption that scaling in Physical AI follows the same patterns as vision-language modeling. The location of information bottlenecks within the visuomotor pipeline determines which components benefit from scaling investments. When discrete tokenization fixes the information capacity at the action level, even state of the art vision encoders become functionally equivalent to modest convolutional networks.

Looking forward, the field must develop diagnostic tools to map information flow in VLA architectures. We need action representations that preserve the computational advantages of tokenization without imposing hard information ceilings. Whether this requires hierarchical architectures with continuous low level execution and discrete high level planning, or adaptive codebooks that scale capacity with task complexity, remains to be determined. What is clear is