Decoding
π Short definition
Phase where the model generates its answer sequentially, predicting one token (word), adding it to context, then predicting the next, and so on.
π¬ How it works (not magic)
Once the initial prompt has been read (Prefill phase), the model enters a generation loop. To guess the next word, it does not recompute full history: it reads past states stored in the KV Cache. Because it generates one word at a time, the processor often waits for memory (VRAM) to deliver cache and model data.
π‘ Why it matters for on-prem AI
This phase governs the βtokens/sβ throughput users see. Here you hit the Memory Wall: speed is limited by memory pipes, not chip compute power.
π Go deeper
Not clear why itβs slow? Follow this path:
- π§ The Journey of a Prompt (βItβs not magicβ chapter)
- ποΈ Memory bandwidth & the Memory Wall (physical explanation)
- Tokens per second (key metric)
π See also
Inference phase that processes the initial prompt in parallel before word-by-word generation.Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Number of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware.Phase where an already trained LLM produces a response token by token, in two steps (prefill then decoding); it is what sizes on-premise hardware.