InfiniBand
π Short definition
Dedicated low-latency, high-bandwidth network (up to 400β800 Gb/s per link) built for HPC and GPU AI clusters.
π Detailed definition
InfiniBand is both a protocol and physical fabric (cables, switches, HCA cards). It natively supports RDMA β GPUs can send data directly without the CPU.
Generations: HDR (200 Gb/s), NDR (400 Gb/s), XDR (800 Gb/s per port).
InfiniBand vs RoCE:
- InfiniBand: dedicated fabric, natively lossless, lowest latency (~1 Β΅s), high cost.
- RoCE: RDMA on standard Ethernet, cheaper but needs strict network config (PFC/ECN) to avoid packet loss.
π‘ Why it matters for on-prem AI
De facto standard for production GPU clusters (AI datacenters, HPC). Required for full-throughput Tensor Parallelism and Pipeline Parallelism across nodes.
β οΈ Common pitfalls
- Needs dedicated hardware and HPC networking skills.
- Mostly proprietary stack (NVIDIA/Mellanox): hard migration to Ethernet.
- Total cost (switches, cables, HCAs) can exceed GPU server cost.
π See also
Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.Protocol bringing RDMA benefits to standard Ethernet, without a dedicated InfiniBand fabric, at the cost of configuring a lossless network (PFC, ECN).Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.Strategy for distributing an LLM in layer slices across multiple machines.Mechanism letting GPUs exchange data directly with network devices without CPU copies.