Skip to content

Decoding

πŸ“ Short definition

Phase where the model generates its answer sequentially, predicting one token (word), adding it to context, then predicting the next, and so on.

πŸ”¬ How it works (not magic)

Once the initial prompt has been read (Prefill phase), the model enters a generation loop. To guess the next word, it does not recompute full history: it reads past states stored in the KV Cache. Because it generates one word at a time, the processor often waits for memory (VRAM) to deliver cache and model data.

πŸ’‘ Why it matters for on-prem AI

This phase governs the β€œtokens/s” throughput users see. Here you hit the Memory Wall: speed is limited by memory pipes, not chip compute power.

πŸ“š Go deeper

Not clear why it’s slow? Follow this path:

  1. 🧠 The Journey of a Prompt (β€œIt’s not magic” chapter)
  2. 🏎️ Memory bandwidth & the Memory Wall (physical explanation)
  3. Tokens per second (key metric)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.