Speculative Decoding: The Only Lossless Speed Boost — and the One Question That Decides It

TL;DR: Speculative decoding accelerates LLM token generation without changing the output: a small draft model quickly proposes several tokens, and the large model verifies them in a single joint forward pass. The one question that determines its usefulness is: Is my inference memory-bound (in which case the boost is worth it, up to ~2x) or compute-bound (where there is little to no gain)? For most individual users and small batch setups on local hardware, the answer is memory-bound — making the boost real.

What speculative decoding achieves

Speculative decoding separates generation from verification. A small, fast draft model generates k token proposals at once. The large verification model reads all k proposals in a single forward pass and accepts as many of them as match its own probability distribution. From the first deviation onward, it re-decodes from that point.

The result: instead of k sequential steps, it only takes one pass for the draft plus one for the verification. With an acceptance rate of 4 out of 5 proposals, the number of large forward passes is roughly halved.

Why the boost is lossless

Unlike quantization or context truncation, the verification step does not change the output: the acceptance rule is designed so that the resulting output distribution matches that of the large model exactly. That is why the mechanism is titled "Lossless Speedup of Autoregressive Decoding" in the original paper by Levi et al. (2022) — the length of the draft acts as a trade-off between parallelism and acceptance rate.

The one question that determines the gain

The speedup depends on the inference regime. This decision rule is the core of this article:

Rule of thumb: A single prompt on a local GPU (Mac mini, RTX card in a single stream) almost always runs memory-bound — speculative decoding pays off. With fully batched server workloads, the gain shrinks.

Production metrics

The magnitudes shown in the original video and published by Berkeley and Red Hat, respectively:

• Berkeley: ~1.96x speedup for small batches, ~1.21x for larger ones — the gain decreases as batch size increases.
• Red Hat: +27% tokens/s and -19% cost per million tokens with speculative decoding enabled in the vLLM stack.

Fewer steps for the same output directly translate to lower costs per million tokens — making this boost a cost-saving feature as well, not just a latency trick.

Copy-paste vLLM config block

Choose a draft model from the same family (the smallest variant), draft length of 4–8 tokens:

vllm serve Qwen/Qwen3-32B \
  --speculative-config '{"method": "draft", "model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5}'

Alternatively, with a built-in method set (e.g., EAGLE), if the selected model supports it:

# vllm serve Qwen/Qwen3-32B --speculative-config
method: eagle3
num_speculative_tokens: 2

Checklist for your own stack

• Is my setup memory-bound? (Single prompts, small batch → yes.)
• Draft model from the same family as the target model — same tokenizer, better acceptance rate.
• Start num_speculative_tokens between 4 and 8, log the acceptance rate, and adjust accordingly.
• For large batches: measure the gain instead of assuming it; the boost can approach zero.

FAQ

Why is speculative decoding truly lossless even though a smaller model makes the proposals? Because the small model only provides proposals; the final distribution is verified by the large model in a single pass. The acceptance rule mathematically ensures that the resulting output distribution is identical to the sequential output of the large model. Thus, the draft model cannot "dilute" the output — it can only reduce the number of forward passes.

How do I choose the draft model? Pick the smallest variant of the same model family, such as Qwen3-0.6B as a draft for Qwen3-32B. The same family implies the same tokenizer and similar conditional distributions, keeping the acceptance rate high. An unrelated, generic draft model usually leads to more frequent rejections and, consequently, a lower speedup.

Is the boost still worth it for large batches? Yes, but with diminishing returns. In memory-bound scenarios (batch size 1–8), ~1.9x is typical; the more the GPU FLOPs are utilized, the more the gain trends toward 1.2x or less. The reason: verifying the draft tokens occupies the same parallel compute units that are already fully saturated in batch processing.

How are the costs related to this? Fewer large forward passes per generated text directly mean fewer GPU seconds per million tokens — the -19% from Red Hat perfectly illustrates this calculation. With hardware capacity remaining constant, throughput increases, meaning the price per million tokens drops. However, the cost savings are tied to the memory-bound condition — in a full-batch scenario, they shrink alongside the speedup itself.

How can I tell if speculative decoding isn't working for me? Most serving stacks (including vLLM) log the draft acceptance rate per step. An acceptance rate close to the draft length indicates maximum gain; a rate below ~50% means the draft is too long or the model family is too different — reduce the draft length or pick a model from the same family. If the GPU is running at full utilization (large batch), a small gain is normal and not a configuration error.

References

• Levi, G., Fusco, J., Sakr, S., Gerards, M. (2022): Lossless Speedup of Autoregressive Decoding — https://arxiv.org/abs/2203.16487
• Original video on the mechanism: Speculative Decoding: The ONLY Video You Need to Speed Up Inference — https://www.youtube.com/watch?v=_hEGvCwHyH8
• vLLM Documentation on Speculative Decoding — https://docs.vllm.ai/en/latest/features/spec_decode.html
• Berkeley LLM Inference Optimization (speedup numbers) — https://llm-d.ai/
• Red Hat: Benchmark on +27% tokens/s / -19% cost with speculative decoding — https://www.redhat.com/en