LLM inference
π Short definition
Phase where a trained LLM produces a response from a prompt, token by token.
π Detailed definition
Training (learning weights on large corpora) is distinct from inference (running the model to answer). In on-prem AI, local inference dominates: Ollama, llama.cpp, vLLM, etc.
A request splits into two phases: Prefill (processing the prompt) then Decoding (autoregressive generation).
π‘ Why it matters for on-prem AI
All hardware sizing (RAM, VRAM, bandwidth, tokens/s, TTFT) targets inference, not training. The Memory Wall bottleneck appears mainly in decoding.
β οΈ Common pitfalls
- Confusing advertised TFLOPS with real generation throughput.
- Forgetting inference and training have different hardware constraints.
π Go deeper
Want inference step by step?
- π§ The Journey of a Prompt (full cycle: tokenisation β prefill β decoding)
- ποΈ Memory bandwidth (why memory caps generation speed)
- βοΈ Inference engines (Ollama, vLLM, TensorRT-LLM)
π See also
AI model trained on very large text corpora, most often of the Transformer type; locally, its performance depends as much on memory as on the model itself.Inference phase that processes the initial prompt in parallel before word-by-word generation.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.Number of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware.Time To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric.Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.