TTFT
π Short definition
Time between sending a request and receiving the first response token.
π Detailed definition
TTFT mainly reflects prefill latency, context initialization, and runtime overhead. It complements tokens/s: you can have good throughput but a poor start.
π‘ Why it matters for on-prem AI
For interactive use, perceived quality depends heavily on TTFT.
β οΈ Common pitfalls
- Measuring only tokens/s and ignoring time before the first word.
- Comparing TTFT without stating input prompt length.
π See also
Inference phase that processes the initial prompt in parallel before word-by-word generation.Number of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware.Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.