HBM
π Short definition
Memory technology stacking several DRAM dies vertically for bandwidth far above consumer GDDR β reserved for professional accelerators (H100, MI300X, TPU).
π Detailed definition
Where a consumer RTX 4090 offers ~1,008 GB/s via GDDR6X, HBM3 reaches:
| Chip | Memory | Bandwidth |
|---|---|---|
| NVIDIA H100 SXM5 | HBM3 80 GB | 3.35 TB/s |
| AMD MI300X | HBM3 192 GB | 5.3 TB/s |
| Apple M5 Ultra | LPDDR5X 512 GB | 1,200 GB/s |
| RTX 4090 (consumer) | GDDR6X 24 GB | ~1,008 GB/s |
That gap explains why the Memory Wall is less constraining on an H100: it can sustain 70B generation throughput without being as memory-bound as a workstation card.
HBM is stacked on the die substrate via TSV (Through-Silicon Via), shortening electrical paths and lowering latency.
π‘ Why it matters for on-prem AI
HBM appears in datacenter chips β out of typical on-prem budgets except in Scenario D. Knowing these numbers calibrates expectations: a rented H100 at β¬3/h is not comparable to a β¬2,000 RTX 4090 even if both βdo AI.β
β οΈ Common pitfalls
- Confusing HBM and GDDR VRAM in comparisons: bandwidth orders of magnitude differ.
- Treating HBM as βjust better GDDRβ: it is 3D stacking with radically different thermal and cost constraints.
π Go deeper
π See also
Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Very high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading.Amount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be.