VRAM
π Short definition
Very high-bandwidth memory attached to the GPU (e.g. GDDR7).
π Detailed definition
VRAM holds weights, activations, KV Cache, and buffers for models running on the GPU. It delivers far higher throughput than system RAM:
| Memory | Type | Typical throughput |
|---|---|---|
| DDR5 (PC RAM, dual-channel) | SDRAM | ~80β100 GB/s |
| GDDR7 (RTX 50xx) | GDDR | ~1,700 GB/s |
| HBM3 (H100 SXM) | HBM | ~3,350 GB/s |
VRAM bandwidth is the limiting factor for Decodingβthe principle behind the Memory Wall.
π‘ Why it matters for on-prem AI
VRAM capacity and bandwidth determine which models can run fast without heavy offloading. To estimate VRAM needed: model size (GB) + β20% for KV Cache and activations.
β οΈ Common pitfalls
- Looking only at VRAM capacity and ignoring memory bandwidth.
- On multi-GPU setups without NVLink, VRAM on each GPU is not one unified pool: each GPU only sees its own VRAM directly.
π See also
Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Stacked memory with very high bandwidth, used on professional AI accelerators.System main memory, second choice for LLM inference when VRAM is insufficient.Architecture where CPU, GPU and sometimes NPU share a single memory pool, avoiding some copies over PCIe; found on Apple Silicon and some recent AMD APUs.