Adaptive Block-Scaled Data Types: Letting Each Group Choose Its Own Number Format

The push toward 4-bit training and inference for large language models has produced a curious situation. We now have hardware that can multiply 4-bit floating point numbers twice as fast as 8-bit numbers and four times as fast as 16-bit numbers, but the formats we feed that hardware are still surprisingly rigid. A 4-bit format can represent exactly 16 values, and once you commit to FP4, those values are fixed: ±{0, 0.5, 1, 1.5, 2, 3, 4, 6}. The paper "Adaptive Block-Scaled Data Types" by Cook and colleagues at MIT and NVIDIA argues that this rigidity is the source of avoidable error, and that we can do better by letting each small block of values decide for itself how it wants to be represented.

This is a clean idea with a satisfying property: it costs zero extra bits. I want to walk through why it works, what the measurements show, and where I think it runs into limits.

The Problem With FP4's Error Distribution

The first thing to understand is that FP4 spaces its representable points logarithmically. The gap between consecutive small values (0 to 0.5 to 1) is tight, while the gap between large values (4 to 6) is enormous. This is fine if your data is concentrated near zero, but it is exactly wrong for the values that matter most in a quantized group.

Block-scaled formats like NVFP4 scale a group of 16 values by a shared FP8 (E4M3) scale factor, then quantize each value to FP4. Because the scale is chosen so the group's maximum lands near the top of FP4's range, the near-maximal values in each group fall into that wide gap between 4 and 6. These are also frequently the outlier-adjacent values that carry disproportionate signal. So NVFP4 concentrates its worst quantization error precisely where the model is most sensitive.

The earlier Four Over Six (4/6) work tried to patch this by adaptively scaling some groups to a maximum of 4 instead of 6. Scaling to 4 reshapes the error distribution to favor near-maximal values, which helps for more uniformly distributed data. But the fix is awkward. Scaling to 4 means sacrificing the 6 and -6 representable values, leaving you with effectively fewer usable levels in a format that already only has 16. It also requires multiplying the group scale by 1.5, which forces a reduction in the global tensor scale to avoid overflow, shrinking dynamic range.

The insight in this paper is that the real lever is not which maximum value you scale to, but the shape of the error distribution itself. And the most uniform error distribution you can get from 4 bits is simply INT4, with its evenly spaced levels.

IF4: Per-Group Selection Encoded for Free

IF4 (Int/Float 4) gives each group of 16 values a binary choice: store them as FP4, or store them as scaled INT4. The selection is recorded in a place that costs nothing. In NVFP4, every individual value carries its own sign bit, which means the shared FP8 scale factor never needs to be negative. Its sign bit is dead weight. IF4 repurposes that unused sign bit as an indicator: 0 means the group is FP4, 1 means the group is scaled INT4.

This is the part I find elegant. There is no memory overhead, no extra metadata stream, no change to group size. The format reuses a bit that NVFP4 was already paying for and getting nothing from.

The practical consequence is that each group can match its representation to its own distribution. Groups with heavy-tailed, near-log-normal value distributions keep FP4 and its fine resolution near zero. Groups that are more uniform, or whose important mass sits near the top of the range, switch to INT4 and get evenly spaced levels there. Notably, IF4 uses a UE4M3 scale factor (the unsigned variant, since the sign bit is now the indicator) and retains a relative dynamic range of 100 percent, matching NVFP4 and beating both NVFP4 with 4/6 (57.1 percent) and NVINT4 (58.3 percent).

The reported mean squared error over a standard normal distribution tells the story compactly:

• MXFP4 (group size 32): 13.2 × 10⁻³
• NVFP4: 9.0 × 10⁻³
• NVFP4 with 4/6: 7.5 × 10⁻³
• NVINT4: 7.4 × 10⁻³
• IF4: 6.2 × 10⁻³

IF4 achieves roughly a 31 percent MSE reduction over NVFP4 while preserving full dynamic range. The authors extend the same trick to IF3 (E2M0 float or scaled integer) and IF6 (6-bit), showing the concept generalizes across bit-widths rather than being a one-off hack for FP4.

Hardware and the Overhead Question

A format that improves accuracy on paper is worthless if it cannot be implemented efficiently, and this is where many clever quantization schemes die. The economics of 4-bit training are brutal: the time saved by doing FP4 matmuls instead of FP8 must exceed the combined overhead of every supporting operation, including the random Hadamard transform, stochastic rounding, and keeping sensitive layers in higher precision. The matmul speedup is the entire budget, and it is easy to spend it all on overhead.

The paper addresses this directly by designing and evaluating an IF4 Multiply-Accumulate (MAC) unit. The claim is that IF4 can be implemented with no memory overhead and minimal computational overhead relative to existing block-scaled formats. This is plausible: the decode path needs to branch on a single indicator bit per group of 16 values, choosing between an FP4 dequantization and an INT4 dequantization. Both are simple operations, and the branch is shared across 16 values so its cost amortizes well. The fact that the authors went to the trouble of building the MAC unit, rather than asserting feasibility, is what makes the contribution credible as a hardware proposal rather than a software-only curiosity.

My Take

What I appreciate most here is the conceptual reframing. The field has spent a lot of effort treating the 16 FP4 codepoints as a fixed constraint and then bolting on transforms (Hadamard rotations, smoothing, stochastic rounding) to make data conform to that constraint. IF4 instead makes the format conform to the data, at the granularity of 16 values, using a bit that was already free. That is a more honest division of labor.

There is also a deeper point lurking here. FP4 versus INT4 is really a question of whether you want fine resolution near zero (log spacing) or uniform resolution across the range (linear spacing). The optimal choice depends on the local distribution, and distributions vary across a tensor. A per-tensor or per-format decision is necessarily a compromise. Per-group adaptive selection is the natural response, and I would not be surprised to see this generalized further, for example to a small per-group lookup index selecting among several codebooks rather than a binary float-or-int switch, if the hardware can afford it.

I do want to raise the limitation that my earlier tweet flagged, because it is the right question to keep asking. Groups of 16 values are small, and the random Hadamard transform is specifically designed to flatten distributions and spread outliers across dimensions. If Hadamard rotation has already made every group look roughly Gaussian and similar to every other group, the marginal benefit of per-group selection should shrink, because there is less variation between groups to exploit. The MSE figures in the paper are computed over N(0,1), which is already the post-rotation regime, and IF4 still wins there. That is encouraging. But the interaction between adaptive selection and aggressive rotation deserves a dedicated ablation: how often does each group actually choose INT4 over FP4 after Hadamard, and does the accuracy gain hold once rotation is applied to both operands during full training?

Conclusion

Adaptive Block-Scaled Data Types make a strong case that the next gains in low-precision LLM training will come not from new transforms layered on top of fixed formats, but from formats flexible enough to adapt at fine granularity. IF4 delivers lower quantization error and full dynamic range at zero bit cost, generalizes to other bit-widths, and comes with a hardware path. The open questions are about interaction effects: how adaptive selection composes with rotation and stochastic rounding in a complete training recipe, and whether the binary float-or-int choice is the right point on the spectrum or merely the first one. If the hardware overhead really is as small as claimed, this is the kind of design that next-generation accelerators should seriously consider building in.