Pipeline Parallelism
π Short definition
Splitting a model into layer blocks across multiple nodes: each node processes its slice and passes the result to the next.
π Detailed definition
Machine A computes layers 1 to N, sends the result (activations) over the network to machine B, which computes the next layers, and so on until the last node. This is one of the two strategies of Exo on desktop clusters (the other being Tensor Parallelism over Thunderbolt 5 RDMA).
Key difference from Tensor Parallelism:
- Pipeline Parallelism: split between layers (needs a fast network but not extreme).
- Tensor Parallelism: split within a layer (needs NVLink or equivalent very fast links).
Pipeline Parallelism increases memory capacity but adds latency at each node boundary (the famous βpipeline bubblesβ β idle time between slices).
π‘ Why it matters for on-prem AI
The most robust option for office clusters without NVLink (Tensor Parallelism is only conceivable there between Macs linked via Thunderbolt 5 RDMA). Key to understanding why TTFT degrades in Scenario C.
β οΈ Common pitfalls
- Pipeline bubbles hurt latency, especially on long prompts (prefill).
- Poorly suited to massive concurrent requests.
- A slow network (Wi-Fi, 1 GbE) can negate all capacity gains.
π Go deeper
- π AI Clustering: Exo and Ray (how Exo uses this strategy)
- π₯οΈ Scenario C: Desktop Cluster (TTFT impact in practice)
- Tensor Parallelism (the alternative strategy for NVLink nodes)