Skip to content

Pipeline Parallelism

πŸ“ Short definition

Splitting a model into layer blocks across multiple nodes: each node processes its slice and passes the result to the next.

πŸ“– Detailed definition

Machine A computes layers 1 to N, sends the result (activations) over the network to machine B, which computes the next layers, and so on until the last node. This is one of the two strategies of Exo on desktop clusters (the other being Tensor Parallelism over Thunderbolt 5 RDMA).

Key difference from Tensor Parallelism:

  • Pipeline Parallelism: split between layers (needs a fast network but not extreme).
  • Tensor Parallelism: split within a layer (needs NVLink or equivalent very fast links).

Pipeline Parallelism increases memory capacity but adds latency at each node boundary (the famous β€œpipeline bubbles” β€” idle time between slices).

πŸ’‘ Why it matters for on-prem AI

The most robust option for office clusters without NVLink (Tensor Parallelism is only conceivable there between Macs linked via Thunderbolt 5 RDMA). Key to understanding why TTFT degrades in Scenario C.

⚠️ Common pitfalls

  • Pipeline bubbles hurt latency, especially on long prompts (prefill).
  • Poorly suited to massive concurrent requests.
  • A slow network (Wi-Fi, 1 GbE) can negate all capacity gains.

πŸ“š Go deeper

  1. 🌐 AI Clustering: Exo and Ray (how Exo uses this strategy)
  2. πŸ–₯️ Scenario C: Desktop Cluster (TTFT impact in practice)
  3. Tensor Parallelism (the alternative strategy for NVLink nodes)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.