Ray
π Short definition
Open-source Python orchestrator to deploy and scale AI applications across multiple servers, often paired with vLLM.
π Detailed definition
Ray uses a Master/Worker architecture on server farms. Paired with vLLM, it orchestrates Tensor Parallelism (matrix split within a node) and Pipeline Parallelism (split across nodes). It also enables Prefill/Decode disaggregation: one server dedicated to prompt computation (Prefill), another to generation (Decoding).
Contrast with Exo: Ray targets enterprise production (monitoring, fault tolerance, smart routing); Exo targets desktop/homelab.
π‘ Why it matters for on-prem AI
De facto standard for sovereign datacenters. The only architecture that guarantees SLA, high concurrency, and fault tolerance on a multi-node GPU cluster.
β οΈ Common pitfalls
- Complex to operate: requires configured AI networking (RoCE/InfiniBand), shared storage, HPC skills.
- Leaving a multi-node cluster without authentication: since Ray 2.59 (2 October 2026), token authentication is on by default only for local clusters; in Q4 2026 (Ray 2.60 included), a remote or multi-node cluster stays open until Ray 2.61 is released, unless
RAY_AUTH_MODE=tokenis set on every node1. Ray 2.61 will require the same token on every node and every client: generate it (ray get-auth-token --generate) and distribute it before the update, otherwise nodes and clients will no longer be able to connect to the cluster1. - Unnecessary and oversized for office or SMB scenarios.
π Go deeper
- π AI Clustering: Exo and Ray (full chapter)
- π’ Scenario D: Datacenter (Ray + vLLM + RoCE blueprint)
- Tensor Parallelism (intra-node strategy used by Ray)
π See also
Footnotes
-
Ray Project, Release ray-2.59.0 (βtoken authentication by default for local clusters β¦ Remote and multi-node clusters are unchangedβ), 2026-10-02; Ray Docs, Token authentication (βRay 2.61 extends the default to all clustersβ), consulted on 2026-10-10. https://github.com/ray-project/ray/releases/tag/ray-2.59.0 Β· https://docs.ray.io/en/latest/ray-security/token-auth.html β© β©2