Prefill
π Short definition
Initialization phase where the model βreadsβ and encodes the entire user prompt before it starts writing its response.
π¬ How it works (not magic)
The model does not read text; it ingests a large grid of numbers (tokens converted to embeddings). During Prefill, the model cross-relates all those tokens in parallel (via the Attention mechanism) to understand context, and saves intermediate computations in memory. That requires massive compute.
π‘ Why it matters for on-prem AI
Prefill directly affects the initial responsiveness feeling called TTFT (Time To First Token). If the prompt contains 100 pages of text, an undersized server can block for several minutes before showing the first word.
π Go deeper
Not clear on the difference from generation? Follow this path:
- π§ The Journey of a Prompt (the βitβs not magicβ chapter)
- πΎ KV Cache (what is stored during this phase)
- ποΈ Memory Bandwidth (hardware impact)
π See also
Time To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric.Phase where an already trained LLM produces a response token by token, in two steps (prefill then decoding); it is what sizes on-premise hardware.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.