RDMA
π Short definition
Network technique to read/write remote memory with low CPU overhead.
π Detailed definition
In a classic network transfer (TCP/IP), data passes through the CPU (copy user space β kernel β NIC). RDMA bypasses that path: the NIC accesses application memory directly, with ~1β2 Β΅s latency and near-zero CPU load.
Two main implementations:
- InfiniBand: dedicated fabric, natively lossless, HPC/high-performance AI standard.
- RoCE: RDMA over Ethernet, lower cost but requires lossless configuration (PFC/ECN).
With GPUDirect RDMA, DMA goes further: the NIC accesses GPU VRAM directly, without passing through CPU RAM.
π‘ Why it matters for on-prem AI
As soon as AI workloads are distributed across multiple nodes, RDMA is what separates an efficient cluster from one limited by network latency.
β οΈ Common pitfalls
- Comparing RDMA and TCP/IP without considering CPU load and microsecond latency.
- Overestimating RDMA impact on small workloads that are mostly compute-bound.
π See also
Dedicated high-performance network fabric for GPU clusters, HPC and AI datacenter standard.Protocol bringing RDMA benefits to standard Ethernet, without a dedicated InfiniBand fabric, at the cost of configuring a lossless network (PFC, ECN).Mechanism letting GPUs exchange data directly with network devices without CPU copies.Architecture using several GPUs in one machine or a cluster to increase memory and throughput; the gains depend on the interconnect (PCIe, NVLink).