Unit measuring floating-point compute capacity per second.
π Detailed definition
TFLOPS is useful for comparing raw compute potential, but it is not a sufficient predictor of LLM performance.
In autoregressive inference, memory and data movement can dominate.
π‘ Why it matters for on-prem AI
Avoids sizing mistakes: a machine very strong in TFLOPS can still be slow if memory bandwidth is low.
Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Amount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be.
A guide by Damien BECHERINI, written with the help of AI agents. The author reviews and approves every change before publication, and every figure cites its primary source. Content dedicated to the public domain (CC0 1.0). Β· Method Β· Licence