Tensor Parallelism
π Short definition
Matrix operations for a single layer are split and computed in parallel across several GPUsβthey act as one giant GPU.
π Detailed definition
Unlike Pipeline Parallelism (splitting between layers), the split here is within a single layer: each GPU computes part of the matrix, then GPUs synchronize partial results over the interconnect.
That synchronization happens very often (every layer), so it needs an extremely fast interconnect: NVLink or NVSwitch on HGX nodes. On PCIe alone, insufficient bandwidth wipes out the benefit.
Used natively by vLLM and TensorRT-LLM. Setting tp=8 on an 8-GPU HGX node spreads the model across all eight chips.
π‘ Why it matters for on-prem AI
Reduces generation latency in datacenter production: each GPU only reads 1/N of the weights, but the per-layer synchronizations (all-reduce via NCCL) make the gain sub-linear. It stays below ΓN, and all the more so when the interconnect is slow. Key to Scenario D: lets the eight GPUs on an HGX node act as one accelerator.
β οΈ Common pitfalls
- Inefficient on PCIe alone (bus too slow for inter-GPU sync).
- Requires the number of attention heads to be divisible by TP.
- Does not apply across nodes without pairing with Pipeline Parallelism.
π Go deeper
- π’ Scenario D: Datacenter (Tensor Parallelism + NVLink + Ray in practice)
- π AI Clustering: Exo and Ray (comparison with Pipeline Parallelism)
- π§© Multi-GPU Workstations (why TP plateaus on PCIe and is only efficient with the NVLink / NVSwitch of SXM servers)