Tokens per second
π Short definition
Number of tokens the model generates per second during the generation phase.
π Detailed definition
This metric depends on the model, quantization, inference engine, batch size, and hardware. It is a practical indicator of user-facing βsmoothness,β but it must be read together with TTFT.
π‘ Why it matters for on-prem AI
The most intuitive metric for comparing hardware setups in conversational use.
π See also
Time To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.Amount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be.