Skip to content

VRAM

πŸ“ Short definition

Very high-bandwidth memory attached to the GPU (e.g. GDDR7).

πŸ“– Detailed definition

VRAM holds weights, activations, KV Cache, and buffers for models running on the GPU. It delivers far higher throughput than system RAM:

MemoryTypeTypical throughput
DDR5 (PC RAM, dual-channel)SDRAM~80–100 GB/s
GDDR7 (RTX 50xx)GDDR~1,700 GB/s
HBM3 (H100 SXM)HBM~3,350 GB/s

VRAM bandwidth is the limiting factor for Decodingβ€”the principle behind the Memory Wall.

πŸ’‘ Why it matters for on-prem AI

VRAM capacity and bandwidth determine which models can run fast without heavy offloading. To estimate VRAM needed: model size (GB) + β‰ˆ20% for KV Cache and activations.

⚠️ Common pitfalls

  • Looking only at VRAM capacity and ignoring memory bandwidth.
  • On multi-GPU setups without NVLink, VRAM on each GPU is not one unified pool: each GPU only sees its own VRAM directly.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.