GPUDirect RDMA
π Short definition
NVIDIA technology letting a GPU send or receive data directly from/to a compatible NIC without CPU memory (zero-copy).
π Detailed definition
In a classic pipeline, GPU β network requires: GPU VRAM β CPU RAM β NIC (two copies). GPUDirect RDMA removes the CPU copy by letting the NIC access GPU VRAM directly over PCIe (DMA-BUF).
Used by NCCL for inter-node communication in distributed clusters.
π‘ Why it matters for on-prem AI
Cuts latency and CPU load during inter-GPU sync on InfiniBand or RoCE. Critical to sustain Tensor Parallelism and Pipeline Parallelism throughput at scale.
β οΈ Common pitfalls
- Fails if GPU and NIC sit on separate PCIe branches with unfavorable NUMA topology.
- Without DMA-BUF or a loaded
nvidia-peermem, NCCL silently falls back to CPU copy: check in the logs (NCCL_DEBUG=INFO) that theNET/IB/GDRDMApath is actually used.
π See also
Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.Strategy for distributing an LLM in layer slices across multiple machines.Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.NVIDIA collective communication library optimized for large-scale GPU-to-GPU transfers.Dedicated high-performance network fabric for GPU clusters, HPC and AI datacenter standard.Standard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink.