Skip to content

Memory Wall

πŸ“ Short definition

Situation where memory throughput limits performance more than raw compute power.

πŸ“– Detailed definition

In modern AI workloads, especially autoregressive generation, compute units often wait for data. When the arithmetic intensity ratio turns unfavorable, final throughput drops.

πŸ’‘ Why it matters for on-prem AI

Machine choice is not TFLOPS alone: memory bandwidth, memory type, and interconnects directly affect user experience.

⚠️ Common pitfalls

  • Buying hardware on TFLOPS alone.
  • Ignoring the gap between prefill and decoding performance.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.