Skip to content

HBM

πŸ“ Short definition

Memory technology stacking several DRAM dies vertically for bandwidth far above consumer GDDR β€” reserved for professional accelerators (H100, MI300X, TPU).

πŸ“– Detailed definition

Where a consumer RTX 4090 offers ~1,008 GB/s via GDDR6X, HBM3 reaches:

ChipMemoryBandwidth
NVIDIA H100 SXM5HBM3 80 GB3.35 TB/s
AMD MI300XHBM3 192 GB5.3 TB/s
Apple M5 UltraLPDDR5X 512 GB1,200 GB/s
RTX 4090 (consumer)GDDR6X 24 GB~1,008 GB/s

That gap explains why the Memory Wall is less constraining on an H100: it can sustain 70B generation throughput without being as memory-bound as a workstation card.

HBM is stacked on the die substrate via TSV (Through-Silicon Via), shortening electrical paths and lowering latency.

πŸ’‘ Why it matters for on-prem AI

HBM appears in datacenter chips β€” out of typical on-prem budgets except in Scenario D. Knowing these numbers calibrates expectations: a rented H100 at €3/h is not comparable to a €2,000 RTX 4090 even if both β€œdo AI.”

⚠️ Common pitfalls

  • Confusing HBM and GDDR VRAM in comparisons: bandwidth orders of magnitude differ.
  • Treating HBM as β€œjust better GDDR”: it is 3D stacking with radically different thermal and cost constraints.

πŸ“š Go deeper

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.