Skip to content

🏭 Scenario D: Datacenter (RoCE & Multi-GPU)

Your client is a large enterprise, a sovereign cloud host, or a public institution. The requirements are uncompromising: host a 70B-class model or an MoE giant of 300B to over 1T parameters, and above all, serve dozens or even hundreds of users at the same time with instant response time.

Scenario B (the Appliance) would choke under concurrent load, and Scenario C (Exo Cluster) has a TTFT that is far too slow. For massive production, there is no secret: you must switch to the standard AI datacenter architecture.


🏗️ Hardware Architecture

Here, the basic unit is no longer the graphics card, but the Server Node and the Fabric Network.

  • The Node (Scale-Up): A rack-format server (e.g. NVIDIA HGX architecture) containing 8 datacenter-class GPUs (NVIDIA H200, B200, or B300; AMD Instinct MI350X/MI455X)1. Unlike a classic PC, these 8 chips do not communicate over PCIe, but via NVLink and NVSwitch. This bus lets chips exchange data at 1,800 GB/s (on Blackwell)2.
  • The Network (Scale-Out): To connect several nodes together, very high-throughput network cards (400 Gbps or 800 Gbps) compatible with RDMA are used. The standard is InfiniBand or RoCEv2 (RDMA over Converged Ethernet)3.
  • Storage: Distributed NVMe flash storage accessible via GPUDirect Storage, to load terabytes of model weights in seconds at startup.

Estimated budget (Q4 2026): from about €280,000 (HGX H200 node, $300–420k according to third-party estimates, on quote) to over €1 million per node (GB300 on quote)4, excluding network infrastructure, energy, and cooling costs. An 8× RTX PRO 6000 server (OEM list price $266k, ~€250,000) costs less, but its GPUs communicate over PCIe, without NVLink or NVSwitch: it is a PCIe variant that does not benefit from the NVLink Tensor Parallelism described below.


⚙️ Software Stack and Mechanism

This hardware extravagance demands inference engines that can exploit it to the millisecond: vLLM (≥ 0.34.0 in Q4 2026, security floor, see Sécurité; vllm serve) or the official TensorRT-LLM SDK (trtllm-serve; in Q4 2026, pin 1.3.0 in production, stable since 20 November 2026, rather than the 1.4 release candidates, and disable the telemetry enabled by default since 1.3 with TRTLLM_NO_USAGE_STATS=1)5. Multi-node orchestration is handled by Ray (Ray Serve LLM, GA since 2.59 with KV-cache-aware routing)6.

The Magic of Tensor Parallelism

On the Mac Cluster (Scenario C), we saw Pipeline Parallelism (layer-by-layer splitting), which increases latency. In an HGX node, the incredible speed of NVLink enables Tensor Parallelism (TP). A single mathematical operation (a matrix) is split and computed at the same time by the 8 GPUs.

  • Result: The 8 cards act as one giant GPU. Generation latency collapses, and tokens/s explode, even on a massive model.

Extreme Formats (FP4)

If you deploy NVIDIA Blackwell (B200) chips, the software will natively use FP4 or FP8 quantization. This lets gigantic models fit in a single 8-GPU node — DeepSeek V4.1 Flash or GLM-5.3 (~760 GB in FP8) in 8× H200, Kimi K3 (~1.5 TB in MXFP4) in 8× B300 or 16× H2007 — avoiding having to cross the RoCE network for every computation8.


The Network Engineering Trap (The RoCE drama)


📋 The Architect’s Verdict

✅ When to use this Blueprint?

  • Production at scale: This is the only viable architecture for serving real sovereign SaaS applications (like an internal enterprise ChatGPT for 1,000 employees).
  • Need for guarantees (SLA): When TTFT must always stay below 500 ms, regardless of the number of connected users.

❌ When to avoid this Blueprint?

  • If you do not have a dedicated network engineer. Operating a RoCE/InfiniBand fabric and a Ray cluster requires very specialized administration skills, often from the High Performance Computing (HPC) world.
  • Datacenter constraints: These machines are giant radiators. A standard rack often cannot cool such a node without heavy upgrades (Direct Liquid Cooling).

Scenario D is the only blueprint that justifies tooling monitoring in production: vLLM metrics, Prometheus + Grafana stack, dashboards and alerts are covered in 📊 Monitoring de la stack d'inférence. Additional elements specific to this blueprint:

GPU and VRAM (per node):

Fenêtre de terminal
# Real-time monitoring of all GPUs
watch -n 1 nvidia-smi
# CSV format for Prometheus export
nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,\
memory.used,memory.free,temperature.gpu,power.draw \
--format=csv -l 5

Network (RoCE/InfiniBand):

Fenêtre de terminal
# RDMA counters (errors, retransmissions)
rdma statistic show
# Packet loss on RoCE interface
ethtool -S <interface> | grep -E "rx_discards|tx_discards"

Storage Wall — SSD→VRAM boot time (MTTR impact)

The “Memory Wall” covers steady-state performance. The “Storage Wall” covers restarts: each vLLM restart requires reloading model weights from SSD to VRAM.

ModelBF16 sizePCIe 3.0 SSD (3 GB/s)PCIe 5.0 NVMe SSD (10 GB/s)GPUDirect Storage
70B~140 GB~47 seconds~14 seconds~8 seconds
405B~810 GB~4.5 minutes~81 seconds~45 seconds

For a datacenter SLA with an MTTR (Mean Time To Recovery) target under 2 minutes, a 405B in BF16 on a PCIe 3.0 SSD is incompatible with that objective. Solutions:

  • PCIe 5.0 NVMe in RAID 0: doubles sequential throughput (~20 GB/s real), MTTR < 45 seconds on a 405B
  • GPUDirect Storage (NVIDIA Magnum IO): direct SSD→VRAM transfer without CPU copy, reduces system load and improves throughput9
  • Quantized model: a 405B in Q4 (~230 GB) reduces load time by ~65% vs BF16
  • vLLM preload: since vLLM 0.31 (October 2026), vllm preload starts a daemon that keeps the already-quantized weights in GPU memory from one engine restart to the next (with a /health endpoint); this shortens service restarts, not a machine reboot or a GPU failure10

🛡️ High Availability and Recovery (DRP)

Blueprint D is the only scenario where a formal disaster recovery plan is economically justified. Two hours of downtime on a production HGX node represents significant operational and reputational cost.

HA strategies by component

ComponentHA strategyNotes
vLLMDual instance with load balancer (Nginx / HAProxy)Rolling restart for zero-downtime updates
Qdrant clusterDistributed mode (3 nodes minimum) with replicationQdrant natively supports sharding and replication
PostgreSQL / SQLiteStreaming replication (Postgres) or WAL archivingSQLite insufficient for production D — migrate to Postgres
Model storageNVMe RAID 0 + daily snapshot to NAS or sovereign S3GPUDirect Storage needs a dedicated path — exclude from software RAID
RoCE networkSwitch redundancy (dual-spine) + active PFC/ECN monitoringUncontrolled packet loss silently collapses throughput

RTO / RPO targets

IncidentTarget RTOTarget RPOAction
vLLM process crash< 2 min0 (no data loss)Automatic restart (systemd / Docker restart policy)
Single GPU failure (8 GPU)< 5 min0Tensor Parallelism reduced to 7 GPUs until replacement
Full node failure< 30 min< 15 minFailover to pre-configured standby node
Vector DB corruption< 1 h< 1 hRestore from latest Qdrant snapshot
Site disaster (fire, flood)< 4 h< 24 hOff-site replication (secondary datacenter or sovereign cloud)

Minimum daily backup

Fenêtre de terminal
# Qdrant snapshot (all collections)
curl -X POST http://localhost:6333/snapshots
# Postgres export (histories, agent metadata)
pg_dump -Fc ia_on_prem_db > /backup/$(date +%F)_pg.dump
# Off-site copy (rsync to secondary NAS example)
rsync -az /backup/ nas-secondary:/ia-on-prem-backup/

📚 Sources and References

Footnotes

  1. NVIDIA Newsroom, NVIDIA Announces Financial Results for Second Quarter Fiscal 2027 (Blackwell Ultra B300 / GB300 shipping; Vera Rubin “in full production”, allocated first to the large clouds), 26 August 2026. https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-second-quarter-fiscal-2027 · AMD, AAI 2026: AMD Delivers Full-Stack Compute for the Agentic AI Era (MI350X available, Helios / MI455X rack “in production”, first racks 2H 2026), 23 July 2026. https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era ↩

  2. NVIDIA Technical Blog, NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Language Model Inference (HGX architecture, Blackwell NVLink 1.8 TB/s), 2024-2026. https://developer.nvidia.com/blog/nvidia-nvlink-and-nvidia-nvswitch-supercharge-large-language-model-inference/ ↩

  3. NVIDIA, RDMA over Converged Ethernet - RoCE | Cumulus Linux (Critical importance of PFC/ECN to avoid LLM performance collapse), 2026. https://docs.nvidia.com/networking-ethernet-software/cumulus-linux/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/ ↩ ↩2

  4. AMD, AAI 2026 — note 8 (8× RTX PRO 6000 server at an OEM list price of $265,928 as of 2026-07-16; MI350P server estimated at $327,238), 23 July 2026. https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era · NVIDIA HGX H200 and DGX B300: no public list price, third-party estimates of $300–420k (H200) and ≈ US$400k (B300), on quote. ↩

  5. NVIDIA, TensorRT-LLM — Release v1.3.0 (first stable release of the 1.3 line after 30 RCs; usage telemetry enabled by default, can be disabled with TRTLLM_NO_USAGE_STATS=1), 20 November 2026. https://github.com/NVIDIA/TensorRT-LLM/releases/tag/v1.3.0 · PyPI, tensorrt-llm (1.4.0rc0 → 1.4.0rc4, October–November 2026), captured 2026-11-19. https://pypi.org/project/tensorrt-llm/ · NVIDIA, TensorRT-LLM Release Notes (TensorRT backend removed in 1.2, PyTorch by default, trtllm-serve). https://nvidia.github.io/TensorRT-LLM/release-notes.html · vLLM Project, Release v0.34.0 (security floor in Q4 2026: fixes advisory GHSA-xgh8-qwjh-q853 and includes the fixes of 0.33.0), 16 November 2026. https://github.com/vllm-project/vllm/releases/tag/v0.34.0 ↩

  6. Ray Project, Release ray-2.59.0 (Ray Serve LLM GA: KV-aware routing, prefill/decode disaggregation, vLLM pinned at 0.27.0), 2 October 2026. https://github.com/ray-project/ray/releases/tag/ray-2.59.0 ↩

  7. DeepSeek AI, DeepSeek-V4.1-Flash (≈ 763 GB in FP8, MIT), September 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash · Moonshot AI, Kimi K3 (native MXFP4 weights, ≈ 1.4–1.56 TB), July 2026. https://huggingface.co/moonshotai/Kimi-K3 ↩

  8. NVIDIA Technical Blog, Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache (Blackwell, native TensorRT-LLM), December 2025. https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/ ↩

  9. NVIDIA, GPUDirect Storage Overview (direct NVMe→VRAM transfer, no CPU copy, Magnum IO). https://developer.nvidia.com/gpudirect-storage ↩

  10. vLLM Project, Release v0.31.0 (vllm preload: daemon that keeps post-quantization weights in GPU memory across engine restarts, #56680, #58552), 2026-10-05. https://github.com/vllm-project/vllm/releases/tag/v0.31.0 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.