Skip to content

Tensor Parallelism

πŸ“ Short definition

Matrix operations for a single layer are split and computed in parallel across several GPUsβ€”they act as one giant GPU.

πŸ“– Detailed definition

Unlike Pipeline Parallelism (splitting between layers), the split here is within a single layer: each GPU computes part of the matrix, then GPUs synchronize partial results over the interconnect.

That synchronization happens very often (every layer), so it needs an extremely fast interconnect: NVLink or NVSwitch on HGX nodes. On PCIe alone, insufficient bandwidth wipes out the benefit.

Used natively by vLLM and TensorRT-LLM. Setting tp=8 on an 8-GPU HGX node spreads the model across all eight chips.

πŸ’‘ Why it matters for on-prem AI

Reduces generation latency in datacenter production: each GPU only reads 1/N of the weights, but the per-layer synchronizations (all-reduce via NCCL) make the gain sub-linear. It stays below Γ—N, and all the more so when the interconnect is slow. Key to Scenario D: lets the eight GPUs on an HGX node act as one accelerator.

⚠️ Common pitfalls

  • Inefficient on PCIe alone (bus too slow for inter-GPU sync).
  • Requires the number of attention heads to be divisible by TP.
  • Does not apply across nodes without pairing with Pipeline Parallelism.

πŸ“š Go deeper

  1. 🏒 Scenario D: Datacenter (Tensor Parallelism + NVLink + Ray in practice)
  2. 🌐 AI Clustering: Exo and Ray (comparison with Pipeline Parallelism)
  3. 🧩 Multi-GPU Workstations (why TP plateaus on PCIe and is only efficient with the NVLink / NVSwitch of SXM servers)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.