Skip to content

Speculative Decoding

πŸ“ Short definition

Inference acceleration technique where a small fast model (β€œdraft model”) generates candidate tokens ahead of time, then a large model (β€œtarget model”) verifies and corrects the batch in a single parallel pass.

πŸ“– Detailed definition

The bottleneck in LLM generation is its autoregressive nature: each token must wait for the previous one before being computed. Speculative decoding works around this constraint in two phases:

  1. Draft phase: a lightweight model (e.g. a 1B or 7B) generates Ξ³ candidate tokens in Ξ³ fast passes.
  2. Verification phase: the large target model evaluates the Ξ³ candidates in a single parallel forward pass β€” as fast as generating 1 token. It accepts valid tokens and rejects incorrect ones from the first mismatch.

If the draft model correctly predicts k tokens out of Ξ³, you get k+1 tokens at the cost of a single call to the large model.

Draft model (7B) β†’ [tok1, tok2, tok3, tok4] (4 candidate tokens)
↓
Target model (70B) β†’ βœ… tok1 βœ… tok2 ❌ tok3 [stop, correction]
β†’ Result: tok1, tok2, tok3-corrected (3 tokens in 1 pass)

Efficiency condition

Real gains depend on the acceptance rate of candidate tokens. When the draft model correctly predicts 70 to 90% of tokens (strong distribution correlation with the target model), acceleration is significant. If the acceptance rate is low (< 50%), frequent verifications cancel the benefit.

βš™οΈ Support in vLLM

vLLM supports speculative decoding via the --speculative-config parameter (JSON object) at server startup; the former --speculative-model flag no longer appears in the documentation as of 2026-10-10 1:

FenΓͺtre de terminal
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--speculative-config '{"method": "draft_model", "model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 5}'

The num_speculative_tokens key (Ξ³) controls how many tokens the draft model generates at each step. A value between 3 and 8 is generally recommended per vLLM docs.

πŸ’‘ Why it matters for on-prem AI

For a sovereign stack with a single 70B model in service, speculative decoding can reduce the generation time perceived by users (inter-token latency) without changing hardware β€” provided you have the memory capacity for the second model. It does not improve TTFT, which depends on prefill, and may even lengthen it slightly.

It is particularly attractive on high-capacity blueprints (128 GB+ unified memory) where loading a 1B–7B draft model adds only a few extra gigabytes.

⚠️ Common pitfalls

  • Do not load the same model twice. The draft model must be a different, smaller model aligned with the target.
  • Does not improve total throughput (multi-user). The gain is on per-request latency. Under heavy concurrent load, PagedAttention and continuous batching remain priorities.
  • Incompatible with some quantization modes. Very aggressive quantizations (Q2) can degrade acceptance rate and cancel the benefit.
  • Gain figures: highly variable. Do not cite a speed multiplier without specifying the model, draft model, measured acceptance rate, and batch size.

πŸ“š To understand in depth

  1. βš™οΈ Inference Engines β€” vLLM ( --speculative-config parameter, configuration)
  2. 🧠 APU & Unified Memory (hardware prerequisite: 128 GB+)
  3. πŸ’Ύ KV Cache & Context (understanding why autoregressivity is the bottleneck)

πŸ”— See also

πŸ“š Sources

Footnotes

  1. vLLM Project, Speculative Decoding β€” official documentation (--speculative-config with the method, model, num_speculative_tokens keys; draft model, EAGLE, MTP, n-gram variants), accessed 2026-10-10. https://docs.vllm.ai/en/stable/features/speculative_decoding/ ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.