Skip to content

Multi-GPU

πŸ“ Short definition

Architecture using several GPU cards in one machine or cluster.

πŸ“– Detailed definition

Multi-GPU increases total memory capacity, throughput, or both. Gains depend directly on inter-GPU links:

TopologyInter-GPU bandwidthTypical use
PCIe x16~64 GB/s (Gen5)desktop builds, Exo
NVLink (H100)900 GB/spro/datacenter nodes
NVLink 5 + NVSwitch (HGX B200)1,800 GB/s per GPU (~14.4 TB/s aggregate across 8 GPUs)HGX B200 nodes

Two parallelism strategies use multi-GPU: Tensor Parallelism (splits work within a layer) and Pipeline Parallelism (splits layers across nodes).

πŸ’‘ Why it matters for on-prem AI

Required for models exceeding one GPU’s VRAM (e.g. Llama 3.1 405B in FP8 = ~405 GB of weights, at 1 byte per parameter). Interconnect speed is often more limiting than raw compute.

⚠️ Common pitfalls

  • Several GPUs’ VRAM is not one unified pool on PCIe: each GPU reaches another’s weights over the bus with latency.
  • On PCIe, Tensor Parallelism is inefficient: prefer Pipeline Parallelism to reduce exchange frequency.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.