Skip to content

Memory bandwidth

πŸ“ Short definition

Data transferred per second between memory and CPU/GPU, usually in GB/s.

πŸ“– Detailed definition

Higher bandwidth lets the model be β€œre-read” faster during generation. Quick local inference estimate: theoretical tokens/s β‰ˆ memory GB/s divided by model size in GB.

πŸ’‘ Why it matters for on-prem AI

Key indicator for predicting generation smoothness on large LLMs.

⚠️ Common pitfalls

  • Treating the tokens/s formula as a guaranteed real value.
  • Confusing local memory bandwidth with network bandwidth.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.