Memory bandwidth
π Short definition
Data transferred per second between memory and CPU/GPU, usually in GB/s.
π Detailed definition
Higher bandwidth lets the model be βre-readβ faster during generation. Quick local inference estimate: theoretical tokens/s β memory GB/s divided by model size in GB.
π‘ Why it matters for on-prem AI
Key indicator for predicting generation smoothness on large LLMs.
β οΈ Common pitfalls
- Treating the tokens/s formula as a guaranteed real value.
- Confusing local memory bandwidth with network bandwidth.
π See also
Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Number of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware.