Skip to content

LLM inference

πŸ“ Short definition

Phase where a trained LLM produces a response from a prompt, token by token.

πŸ“– Detailed definition

Training (learning weights on large corpora) is distinct from inference (running the model to answer). In on-prem AI, local inference dominates: Ollama, llama.cpp, vLLM, etc.

A request splits into two phases: Prefill (processing the prompt) then Decoding (autoregressive generation).

πŸ’‘ Why it matters for on-prem AI

All hardware sizing (RAM, VRAM, bandwidth, tokens/s, TTFT) targets inference, not training. The Memory Wall bottleneck appears mainly in decoding.

⚠️ Common pitfalls

  • Confusing advertised TFLOPS with real generation throughput.
  • Forgetting inference and training have different hardware constraints.

πŸ“š Go deeper

Want inference step by step?

  1. 🧠 The Journey of a Prompt (full cycle: tokenisation β†’ prefill β†’ decoding)
  2. 🏎️ Memory bandwidth (why memory caps generation speed)
  3. βš™οΈ Inference engines (Ollama, vLLM, TensorRT-LLM)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.