Skip to content

Tokens per second

πŸ“ Short definition

Number of tokens the model generates per second during the generation phase.

πŸ“– Detailed definition

This metric depends on the model, quantization, inference engine, batch size, and hardware. It is a practical indicator of user-facing β€œsmoothness,” but it must be read together with TTFT.

πŸ’‘ Why it matters for on-prem AI

The most intuitive metric for comparing hardware setups in conversational use.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.