Skip to content

GPUDirect RDMA

πŸ“ Short definition

NVIDIA technology letting a GPU send or receive data directly from/to a compatible NIC without CPU memory (zero-copy).

πŸ“– Detailed definition

In a classic pipeline, GPU β†’ network requires: GPU VRAM β†’ CPU RAM β†’ NIC (two copies). GPUDirect RDMA removes the CPU copy by letting the NIC access GPU VRAM directly over PCIe (DMA-BUF).

Used by NCCL for inter-node communication in distributed clusters.

πŸ’‘ Why it matters for on-prem AI

Cuts latency and CPU load during inter-GPU sync on InfiniBand or RoCE. Critical to sustain Tensor Parallelism and Pipeline Parallelism throughput at scale.

⚠️ Common pitfalls

  • Fails if GPU and NIC sit on separate PCIe branches with unfavorable NUMA topology.
  • Without DMA-BUF or a loaded nvidia-peermem, NCCL silently falls back to CPU copy: check in the logs (NCCL_DEBUG=INFO) that the NET/IB/GDRDMA path is actually used.

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.