Why Your Speculative Decoder's Gamma Is Probably Wrong After Quantization

Speculative decoding has quietly become one of the most reliable ways to speed up large language model inference without sacrificing output quality. The idea is elegant: pair a small, fast draft model with a larger, more accurate target model. The draft proposes several tokens at once, and the target verifies them in a single forward pass. When the draft guesses correctly, you get multiple tokens for the price of one target step. The math works out to real latency improvements, and the technique is provably lossless with respect to the target distribution.

But there is a hyperparameter sitting at the center of this scheme that almost everyone ignores: the speculation length, conventionally written as gamma. It controls how many tokens the draft proposes per step. Nearly every deployed system hardcodes it to 4 and moves on. A recent paper, SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection by Shikhar Shukla (University of Kentucky), argues that this default is not just suboptimal, but that the optimal value depends on something nobody has been measuring: how aggressively you have compressed your target model.

The coupling between compression and speculation

The core claim of SpecKV is that compression and speculation length are coupled rather than independent. This is the kind of interaction that is easy to overlook because the two techniques live in different parts of the inference stack. Quantization is a memory and throughput optimization applied to model weights or the KV cache. Speculation length is a scheduling decision about how many candidate tokens to draft. They feel orthogonal.

They are not. Quantization, whether INT8 or 4-bit NF4, perturbs the target model's output distribution. Since the acceptance of a draft token depends on whether it matches the target's top prediction, any shift in the target distribution directly changes acceptance probabilities. And acceptance probability is exactly what determines the optimal gamma. If the draft model is highly accurate against the target, a longer speculation pays off because you accept more tokens per step. If the target distribution has drifted under aggressive quantization, the draft's predictions stop matching, acceptance drops, and a long gamma starts wasting target compute on tokens destined for rejection.

This is the mechanism behind the tweet's observation that under NF4 quantization the target distribution moves and a static gamma silently underperforms. The word "silently" is the important one. Nothing crashes. The system keeps producing correct output. You just lose throughput you did not know you had.

What the profiling actually shows

The paper backs the claim with a profiling study rather than a single anecdote. Shukla runs the Llama 3.2 family, using the 1B-Instruct as draft and 3B-Instruct as target, across:

• 4 task categories
• 4 speculation lengths
• 3 compression levels (FP16, INT8, NF4)

This grid produces 240 experiments and 5,112 step-level records, each logging the per-step acceptance rate, draft entropy, and draft confidence. The headline finding from this data is that the optimal gamma shifts across compression regimes. The earlier Smurfs work had already shown that optimal gamma varies by up to 41.2 percent across models, batch sizes, and datasets, but it did not consider compression as a variable. SpecKV adds that axis.

The second finding is the more actionable one. Draft model entropy and confidence correlate with acceptance rate at roughly 0.56. That is a moderate but genuinely useful correlation, and the key property is that both signals are zero cost. The draft model has already computed its output distribution in order to propose tokens. Reading off the entropy and the top-token probability is essentially free. You do not need an auxiliary model or an extra forward pass to predict whether a long speculation is worth attempting. The signal is sitting in the logits you already produced.

This matters because most adaptive inference schemes pay a price for their adaptivity. Early exit mechanisms, mixture-of-experts routing, and model cascades all add decision overhead. SpecKV's controller is a single-layer MLP with 16 hidden units that consumes these free signals and outputs a gamma choice per speculation step. The reported overhead is 0.34 milliseconds per decision, which the paper places at under 0.5 percent of step time.

The results and how to read them

SpecKV reports a 56.0 percent improvement over the fixed gamma=4 baseline, with statistical significance established via a paired bootstrap test at p < 0.001. The objective the MLP optimizes is expected tokens per speculation step, which is the right quantity: it captures both the upside of long speculations when the draft is confident and the downside of wasted target compute when it is not.

A few caveats are worth stating plainly, because the paper itself is honest about its scope.

• Single GPU, single model pair. All experiments run on one RTX 3090 with the 1B draft and 3B target. The 1B-to-3B ratio is fairly small as draft-target pairs go. Many production setups use a much larger target, where acceptance dynamics and the cost balance between draft and target steps differ. The 56 percent figure should be read as a result for this configuration, not a universal constant.

• Tokens per step is a proxy for wall-clock throughput, not identical to it. Expected tokens per step ignores some real-world effects like batching, memory bandwidth pressure under quantized KV caches, and the variance in step latency. A controller that maximizes mean tokens per step could in principle increase latency variance, which matters for tail latency in serving.

• Greedy acceptance. The described verification is greedy top-1 matching. Sampling-based acceptance with the standard speculative rejection criterion would change the acceptance statistics and potentially the entropy-acceptance correlation.

None of these undermine the central insight. They simply mark where the result needs replication before being treated as a deployment default.

My take: the entropy signal points somewhere bigger

The part of this work I find most generative is the entropy-confidence-to-acceptance relationship, and I think it implies more than adaptive gamma. Consider what the 0.56 correlation tells us: the draft model's own uncertainty is a partial readout of how well it agrees with the target. Under compression, that agreement degrades, and the draft entropy reflects it.

This sets up the open question I raised in the original tweet: why stop at selecting gamma? If draft entropy predicts acceptance, then the draft model's calibration relative to the compressed target is the underlying lever. Jointly learning the draft model's temperature alongside gamma could recalibrate the draft to match the compressed target's distribution curvature. A sharper or softer draft distribution would change which tokens it commits to, and a draft that is better aligned to the quantized target could safely support a higher gamma. In effect, you would be co-adapting the proposer and the scheduler instead of treating the proposer as fixed.

There is also a connection worth drawing to the broader adaptive inference literature. SpecKV's per-step gamma selection is structurally a contextual bandit: each step presents a context (the draft signals), you choose an action (gamma), and you observe a reward (accepted tokens). The current MLP is trained offline on profiling data, but an online version that updates as the target's effective distribution drifts, for instance when quantization is applied dynamically or when the workload shifts, would be a natural extension. This would also address distribution shift between profiling time and serving time, which is the usual failure mode of offline-trained controllers.

Conclusion

The contribution of SpecKV is less about the specific 56 percent number and more about naming an interaction that the field had been treating as nonexistent. Compression and speculation length are coupled, the draft model already carries a cheap signal about how that coupling is playing out, and a tiny controller can exploit it for measurable gains. As 4-bit quantization and KV cache compression become standard in production, the assumption that a fixed gamma of 4 transfers across compression levels deserves to be retired. The next interesting steps are joint draft recalibration, online adaptation, and validation across larger draft-target gaps and sampling-based acceptance. The signal is in the logits. We just have to use it.