Skip to content

🖥️ Scenario C: Desktop Cluster (Exo & Thunderbolt)

Scenario B (the Appliance) has a major flaw: its memory is fixed. If your client’s needs evolve and they want to deploy a colossal MoE model of more than 400 billion parameters (requiring more than 300 GB of memory), only a high-end unified-memory workstation can host it — the Mac Studio M5 Ultra, configurable up to 512 GB (1.2 TB/s; option from €25,079 incl. VAT, details in 🧠 APU & Mémoire Unifiée) — and none exceeds 512 GB12.

Before 2025, the only solution was to rent a cloud server or buy a prohibitively expensive datacenter rack. Today, software architecture allows merging several affordable small machines: the Desktop Cluster.


🏗️ Hardware architecture

The idea is to create a compute “farm” sitting on a shelf.

  • Nodes: 4 to 8 compact machines. The standard in 2026 for this scenario is the Mac mini M5 Pro (64 GB unified memory, 307 GB/s, Thunderbolt 5, from €1,999 incl. VAT at 24 GB) — or, pricier but faster, the Mac Studio M5 Max 64 GB (€3,659 incl. VAT) — and recent AMD Ryzen AI Max mini-PCs3.
  • Network: This is the heart of the system. To prevent data transfer from killing performance, machines are connected with Thunderbolt 5 cables (80 Gb/s symmetric, up to 120 Gb/s in asymmetric mode), the only ones that allow the RDMA over Thunderbolt introduced by macOS 26.2. This RDMA (MLX’s JACCL backend) requires a full mesh, with a cable between every pair of machines, which in practice means 2 to 4 Macs (configuration documented by Apple and MLX). Beyond that, or in a daisy chain via a hub, nodes exchange over TCP on Thunderbolt (ring backend, pipeline parallelism), without RDMA — Thunderbolt 4 remains limited to classic IP at 40 Gb/s145.
  • Total capacity: With 4 Mac Minis at 64 GB linked in an RDMA mesh, you get a silent cluster with 256 GB aggregated unified memory; with 6 Mac Minis (384 GB), you must switch to the ring backend over TCP on Thunderbolt, without RDMA.

Estimated budget (Q4 2026): ~€10,000 to €22,000 for 4 to 6 machines (Mac mini M5 Pro 64 GB or Mac Studio M5 Max 64 GB at €3,659 incl. VAT)3. About 10 times cheaper than an equivalent NVIDIA DGX server in VRAM. 64 GB machines follow the DRAM increase (see 💰 Comparaison TCO): prices captured in October 2026 must be re-checked at purchase time.


⚙️ Software stack and mechanism

This hardware miracle is made possible by MLX distributed inference (mlx.launch with a hostfile, pipeline parallelism, RDMA over Thunderbolt 5 since macOS 26.2) — the Exo orchestrator, which popularized the approach must be considered as slowed down in Q4 2026 (status detailed in the AI Clustering chapter)45.

  1. The distributed runtime (MLX, or Exo) installs on all Mac Minis.
  2. They discover each other via the Thunderbolt network.
  3. The massive LLM (e.g. DeepSeek V4-Flash 284B, or V3 671B in 4-bit)6 is split into slices according to Pipeline Parallelism.
  4. Mac #1 computes the first 10 layers of the neural network, sends its raw result via Thunderbolt to Mac #2, which computes the next 10 layers, and so on.

Expected performance

The gain is purely capacity-based: you do not gain speed, you gain the right to run the model. Network latency, even over Thunderbolt, is infinitely slower than internal RAM speed. On a cluster of 8 Mac mini M4 Pro 64 GB running DeepSeek V3 671B in 4-bit, Exo Labs measured 5.4 tokens/s (TTFT 2.9 s, December 2024, without RDMA; the post does not specify the interconnect)7; Apple announces up to 3× better with four Mac Studio M5 linked via RDMA over Thunderbolt 51. On the model side, DeepSeek V4.1 Flash (September 2026) introduces an FP4 KV cache of about 890 bytes per token, which relieves precisely the memory and TTFT constraint of this scenario8.


The latency trap (TTFT)


📋 The architect’s verdict

✅ When to use this blueprint?

  • Frontier model prototyping: For research or engineering teams that must absolutely test monumental LLMs (DeepSeek V4, GLM-5.3, Kimi K3, Qwen3.8-2.4T) without data leaving the company.
  • Background processing: Perfect for asynchronous document analysis (where latency does not matter).
  • Budget scalability: You can start with 2 machines and add a 3rd the following year to increase your VRAM capacity.

❌ When to avoid this blueprint?

  • For a real-time conversational RAG assistant. Waiting 45 seconds for the first word after asking a question about a PDF will frustrate your users.
  • To serve many concurrent collaborators. Thunderbolt networking and Pipeline Parallelism handle massive concurrent requests very poorly. If you need to serve 50 users in real time on a giant model, you must switch to a real datacenter network (RoCE/InfiniBand) and multi-GPU servers — that is the subject of 🏭 Scenario D: Datacenter.

On an Exo cluster, monitoring is more manual than in datacenter production, but a few commands cover the essentials.

On each Mac node:

Fenêtre de terminal
# GPU load and unified memory (macOS)
sudo powermetrics --samplers gpu_power -i 1000 | grep -E "GPU|ANE"
# Thunderbolt network activity
nettop -m tcp -J bytes_in,bytes_out

Via Ollama (if used as frontend):

Fenêtre de terminal
# Status of loaded models
curl http://localhost:11434/api/tags
# Generation metrics in logs
cat ~/.ollama/logs/server.log # macOS ; Linux : journalctl -u ollama --no-pager --follow --pager-end

Key indicators to watch:

MetricAlert thresholdTool
TTFT> 30 s on short promptExo logs
Tokens/s< 2 tok/sExo logs
Unified memory per node> 90 %vm_stat / Activity Monitor
Thunderbolt bandwidth> 70 Gb/s sustainednettop

Storage Wall — model reload time


📚 Sources and references

Footnotes

  1. Apple Newsroom, Apple introduces new Mac Studio with M5 Max and M5 Ultra (M5 Ultra up to 512 GB unified memory, 1.2 TB/s, 512 GB configuration available late October 2026), 2026-08-25. https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ · Apple Store France, Mac Studio (M5 Ultra 96 GB from €6,599 incl. VAT), captured 2026-10-09. https://www.apple.com/fr/shop/buy-mac/mac-studio ↩ ↩2 ↩3 ↩4

  2. MacRumors, Mac Studio With M5 Ultra Now Available With 512GB RAM for $22,299 (512 GB option reserved for the M5 Ultra with 36-core CPU / 80-core GPU, $6,799 + $15,500), 28 October 2026 · MacGeneration, Mac Studio M5 Ultra : l’option 512 Go de RAM coûte plus cher qu’un second Mac Studio (€25,079 incl. VAT, lead times of 13 to 15 weeks), 4 November 2026. https://www.macrumors.com/2026/10/28/mac-studio-m5-ultra-512gb-ram/ · https://www.macg.co/mac/2026/11/mac-studio-m5-ultra-loption-512-go-de-ram-coute-plus-cher-quun-second-mac-studio-312149 ↩

  3. Apple Newsroom, Apple unveils a more powerful Mac mini featuring the all-new M6 and M5 Pro (Mac mini M5 Pro: up to 64 GB, 307 GB/s, Thunderbolt 5), August 2026. https://www.apple.com/newsroom/2026/08/apple-unveils-a-more-powerful-mac-mini-featuring-the-all-new-m6-and-m5-pro/ · Apple Store France, Mac mini (M5 Pro 24 GB €1,999 incl. VAT) and Mac Studio (M5 Max 64 GB €3,659 incl. VAT), captured 2026-10-09. https://www.apple.com/fr/shop/buy-mac/mac-mini ↩ ↩2

  4. Exo Labs, exo — Releases (latest version v1.0.71 of 2026-04-23), consulted 2026-10-09. https://github.com/exo-explore/exo/releases · Exo Labs, GitHub - exo-explore/exo (README: MLX backend only, Linux CPU-only, RDMA Thunderbolt 5 + macOS 26.2), re-read on 2026-10-09. https://github.com/exo-explore/exo ↩ ↩2

  5. Apple MLX, Distributed Communication (mlx.launch and hostfile, JACCL backends — RDMA over Thunderbolt since macOS 26.2, Thunderbolt 5 — and ring), consulted 2026-10-10. https://ml-explore.github.io/mlx/build/html/usage/distributed.html ↩ ↩2 ↩3

  6. DeepSeek AI — Hugging Face organization (V4 family: DeepSeek-V4-Flash 284B-A13B, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro 1.6T; V3 671B retained), captured 2026-10-09. https://huggingface.co/deepseek-ai ↩

  7. Exo Labs, Running DeepSeek V3 671B on M4 Mac Mini Cluster — 12 days of EXO, day 2 (8× Mac mini M4 Pro 64 GB, DeepSeek V3 4-bit, TTFT and tokens/s; undated page, December 2024 series), re-read on 2026-10-09. https://blog.exolabs.net/day-2 ↩

  8. DeepSeek AI, DeepSeek-V4.1-Flash (MIT, 8–16B active, FP4 KV cache 890 bytes/token), September 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.