Skip to content

Prefill

πŸ“ Short definition

Initialization phase where the model β€œreads” and encodes the entire user prompt before it starts writing its response.

πŸ”¬ How it works (not magic)

The model does not read text; it ingests a large grid of numbers (tokens converted to embeddings). During Prefill, the model cross-relates all those tokens in parallel (via the Attention mechanism) to understand context, and saves intermediate computations in memory. That requires massive compute.

πŸ’‘ Why it matters for on-prem AI

Prefill directly affects the initial responsiveness feeling called TTFT (Time To First Token). If the prompt contains 100 pages of text, an undersized server can block for several minutes before showing the first word.

πŸ“š Go deeper

Not clear on the difference from generation? Follow this path:

  1. 🧠 The Journey of a Prompt (the β€œit’s not magic” chapter)
  2. πŸ’Ύ KV Cache (what is stored during this phase)
  3. 🏎️ Memory Bandwidth (hardware impact)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.