Memory Wall
π Short definition
Situation where memory throughput limits performance more than raw compute power.
π Detailed definition
In modern AI workloads, especially autoregressive generation, compute units often wait for data. When the arithmetic intensity ratio turns unfavorable, final throughput drops.
π‘ Why it matters for on-prem AI
Machine choice is not TFLOPS alone: memory bandwidth, memory type, and interconnects directly affect user experience.
β οΈ Common pitfalls
- Buying hardware on TFLOPS alone.
- Ignoring the gap between prefill and decoding performance.
π See also
Amount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.