NCCL
π Short definition
NVIDIA software library handling collective ops (AllReduce, AllGather, Broadcast) across GPUs, using NVLink, InfiniBand, or PCIe per topology.
π Detailed definition
NCCL (pronounced βnickelβ) underpins PyTorch, vLLM, TensorRT-LLM, and Ray for synchronizing gradients or activations across GPUs.
Main operations:
- AllReduce: sum/average a tensor on all GPUs (distributed training and Tensor Parallelism).
- AllGather: each GPU collects fragments from others.
- Broadcast: sends one tensor from a master GPU to all others.
NCCL auto-detects topology (NVLink > InfiniBand > PCIe > Ethernet) and optimizes communication paths.
π‘ Why it matters for on-prem AI
Invisible but critical layer: if NCCL is not configured for NVLink or InfiniBand, all inter-GPU traffic goes over PCIe and sharply degrades Tensor Parallelism throughput.
β οΈ Common pitfalls
NCCL_P2P_DISABLE=1disables direct peer-to-peer transfers β a common misconfiguration that can cut performance fivefold.- NCCL logs show detected topology: always verify NVLink is selected on HGX nodes.
π See also
Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.Dedicated hardware link that connects NVIDIA GPUs at very high speed, bypassing the PCIe bus; it is gone from recent workstation and consumer cards.Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.