Multi-GPU
π Short definition
Architecture using several GPU cards in one machine or cluster.
π Detailed definition
Multi-GPU increases total memory capacity, throughput, or both. Gains depend directly on inter-GPU links:
| Topology | Inter-GPU bandwidth | Typical use |
|---|---|---|
| PCIe x16 | ~64 GB/s (Gen5) | desktop builds, Exo |
| NVLink (H100) | 900 GB/s | pro/datacenter nodes |
| NVLink 5 + NVSwitch (HGX B200) | 1,800 GB/s per GPU (~14.4 TB/s aggregate across 8 GPUs) | HGX B200 nodes |
Two parallelism strategies use multi-GPU: Tensor Parallelism (splits work within a layer) and Pipeline Parallelism (splits layers across nodes).
π‘ Why it matters for on-prem AI
Required for models exceeding one GPUβs VRAM (e.g. Llama 3.1 405B in FP8 = ~405 GB of weights, at 1 byte per parameter). Interconnect speed is often more limiting than raw compute.
β οΈ Common pitfalls
- Several GPUsβ VRAM is not one unified pool on PCIe: each GPU reaches anotherβs weights over the bus with latency.
- On PCIe, Tensor Parallelism is inefficient: prefer Pipeline Parallelism to reduce exchange frequency.
π See also
Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.Strategy for distributing an LLM in layer slices across multiple machines.Dedicated hardware link that connects NVIDIA GPUs at very high speed, bypassing the PCIe bus; it is gone from recent workstation and consumer cards.NVIDIA switch that connects multiple GPUs into a fully non-blocking NVLink fabric inside a node.Standard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink.Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.