Skip to content

Ray

πŸ“ Short definition

Open-source Python orchestrator to deploy and scale AI applications across multiple servers, often paired with vLLM.

πŸ“– Detailed definition

Ray uses a Master/Worker architecture on server farms. Paired with vLLM, it orchestrates Tensor Parallelism (matrix split within a node) and Pipeline Parallelism (split across nodes). It also enables Prefill/Decode disaggregation: one server dedicated to prompt computation (Prefill), another to generation (Decoding).

Contrast with Exo: Ray targets enterprise production (monitoring, fault tolerance, smart routing); Exo targets desktop/homelab.

πŸ’‘ Why it matters for on-prem AI

De facto standard for sovereign datacenters. The only architecture that guarantees SLA, high concurrency, and fault tolerance on a multi-node GPU cluster.

⚠️ Common pitfalls

  • Complex to operate: requires configured AI networking (RoCE/InfiniBand), shared storage, HPC skills.
  • Leaving a multi-node cluster without authentication: since Ray 2.59 (2 October 2026), token authentication is on by default only for local clusters; in Q4 2026 (Ray 2.60 included), a remote or multi-node cluster stays open until Ray 2.61 is released, unless RAY_AUTH_MODE=token is set on every node1. Ray 2.61 will require the same token on every node and every client: generate it (ray get-auth-token --generate) and distribute it before the update, otherwise nodes and clients will no longer be able to connect to the cluster1.
  • Unnecessary and oversized for office or SMB scenarios.

πŸ“š Go deeper

  1. 🌐 AI Clustering: Exo and Ray (full chapter)
  2. 🏒 Scenario D: Datacenter (Ray + vLLM + RoCE blueprint)
  3. Tensor Parallelism (intra-node strategy used by Ray)

πŸ”— See also

Footnotes

  1. Ray Project, Release ray-2.59.0 (β€œtoken authentication by default for local clusters … Remote and multi-node clusters are unchanged”), 2026-10-02; Ray Docs, Token authentication (β€œRay 2.61 extends the default to all clusters”), consulted on 2026-10-10. https://github.com/ray-project/ray/releases/tag/ray-2.59.0 Β· https://docs.ray.io/en/latest/ray-security/token-auth.html ↩ ↩2

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.