🏭 GPU rack servers
After unified-memory APUs (light SMB) and office workstations, GPU rack servers are the standard building block for scenarios B (rackable appliance) and D (datacenter) — see Scenario B and Scenario D.
1. Form factors and GPU capacity
| Format | Typical GPUs | Inference use |
|---|---|---|
| 1U | 1–2 GPU (often RTX/L40S) | SMB, edge, models ≤ 30B quantized |
| 2U | 2–4 PCIe GPUs | SMB / mid-market sweet spot — multi-user vLLM |
| 4U | 4–8 GPU, sometimes intra-node NVLink | 70B+ models, tensor parallelism |
| HGX / OAM / NVL72 | 8× H100/H200/B200/B300, NVSwitch; Vera Rubin (NVL72) in full production since August 2026, first for the large clouds1 | Datacenter, 405B+ models, massive TP |
Constraint #1 remains total addressable VRAM: a Llama 3 70B in FP16 needs ~140 GB of weights alone, not counting concurrent KV Cache.
Outside NVIDIA, three 2026 announcements complete the rack landscape without changing it for an SMB. On the AMD side, the Helios rack (72 Instinct MI455X HBM4, 18 EPYC “Venice”, Pensando networking) has been declared “in production” since 2026-07-23, with Bull, HPE, Lenovo, and Supermicro as integrators, and AMD maintains its second-half 2026 target despite delay rumors: for a sovereign datacenter, it is an alternative to qualify in 2027, not a 2026 purchase23. The Cerebras CS-4 (2026-08-18, three Wafer Scale Engine 3 Turbo, on-chip SRAM, “up to 30×” the inference throughput of a GPU system according to Cerebras) targets neoclouds and hyperscalers, sold direct with no public price4. Finally, Intel is preparing Crescent Island, an inference GPU with 160 GB of LPDDR5X and no published date (see Multi-GPU workstations).
2. Consumer RTX vs datacenter GPU
| Criterion | RTX 5090 / RTX PRO 6000 (workstation/rack 2U) | H100 / H200 / B200 / B300 (HGX) |
|---|---|---|
| VRAM | 32–96 GB GDDR7 | 80–288 GB HBM |
| Memory bandwidth | ~1.8 TB/s5 | ~3–8 TB/s |
| Multi-GPU NVLink | None (no NVLink since Ada; PCIe only)5 | NVSwitch full mesh |
| Acquisition cost | Significantly lower than HGX, but the gap is narrowing with the 2026 rise in GDDR7 GPU prices (RTX PRO 6000 ~$16,000, RTX 5090 ≥ $5,000, see Multi-GPU workstations); an 8× RTX PRO 6000 server is listed at ~$266k by an OEM in July 20262 | Datacenter TCO, enterprise support; no public list price for HGX B300 / GB300 |
| Best for | Scenario B, moderate-load lab | Scenario D, strict SLA, large models |
Previous generations remain safe purchases: NVIDIA AI Enterprise Infra 8.x (May 2026) keeps A100, H100/H200, L40/L40S, and RTX 6000 Ada among supported GPUs; only V100, RTX 4000 SFF Ada, RTX A4000, and Quadro RTX have been retired (still supported on the 7.x LTSB line)6.
3. Quick sizing
Step 1 — Target model weight
Use 4/8-bit quantization and Choosing your model to estimate weight VRAM.
Step 2 — Concurrent KV Cache
Refer to KV Cache and context: each active session consumes VRAM proportional to context length.
Step 3 — Engine
Ollama on a single RTX GPU suits testing; multi-user production switches to vLLM7 or SGLang (CUDA 13 by default since summer 2026); TensorRT-LLM8 has been stable at 1.3.0 since 20 November 2026 (1.4 is in release candidate; usage telemetry is enabled by default and should be disabled on a sovereign deployment)9 — ⚙️ Inference engines.
Step 4 — Multi-node network
Beyond one node, the RoCE or InfiniBand fabric becomes mandatory for inter-server tensor parallelism.
4. Purchase pitfalls
- Underestimating cooling and power: datacenter GPUs in 1U = extreme noise and thermals; plan for a suitable room or rack.
- Shared PCIe x16: check lane splitting when 4 GPUs share the same CPU — see Multi-GPU workstations (same principles).
- Licenses and support: some vendors restrict “datacenter” use of gaming cards — read EULAs before production deployment.
- Forgetting the application front end: the GPU rack serves vLLM; the user UI remains Open WebUI or equivalent.
- Underestimating system RAM: DRAM is still rising every quarter (see 💰 TCO comparison); nodes that offload KV cache or weights to RAM are now also priced in euros per GB.
📋 The architect’s advice
For an SMB wanting a 2U rack × 2× L40S (48 GB, 350 W) or RTX PRO 6000 (96 GB, ~$16,000 in Q3 2026): size for a 32B–70B quantized model + 10–20 concurrent users, Open WebUI front, vLLM behind a reverse proxy — blueprint B.
For a datacenter: start with the target model (405B? 70B dense? MoE?) and work down to the number of HGX GPUs — blueprint D.
📚 Sources
Footnotes
-
NVIDIA Newsroom, NVIDIA Announces Financial Results for Second Quarter Fiscal 2027 (“Vera Rubin, now in full production”, racks at CoreWeave, Google Cloud, Microsoft Azure, OCI, Nebius), 26 August 2026. https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-second-quarter-fiscal-2027 ↩
-
AMD, AAI 2026: AMD Delivers Full-Stack Compute for the Agentic AI Era (Helios “now in production”, 72 MI455X + 18 EPYC Venice, OEMs Bull / HPE / Lenovo / Supermicro; note 8: RTX PRO 6000-based server at OEM list price of $265,928.24 as of 2026-07-16), 23 July 2026. https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era ↩ ↩2
-
Tom’s Hardware, AMD denies report of MI455X delays as Nvidia VR200 systems are rumored to arrive early — company says Helios systems on target for 2H 2026, September 2026. https://www.tomshardware.com/tech-industry/artificial-intelligence/amd-denies-report-of-mi455x-delays-as-nvidia-vr200-systems-are-rumored-to-arrive-early-company-says-helios-systems-on-target-for-2h-2026 ↩
-
Cerebras, Introducing Cerebras CS-4 (three WSE-3 Turbo, “up to 30 times faster inference than GPU systems”, targets neoclouds and hyperscalers, no public price), 18 August 2026. https://www.cerebras.ai/blog/introducing-cerebras-cs-4 ↩
-
NVIDIA, RTX PRO 6000 Blackwell Workstation Edition (96 GB GDDR7 ECC, 1,792 GB/s, PCIe Gen 5, 600 W, no NVLink connector). https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000/ · NVIDIA, GeForce RTX 5090 (32 GB GDDR7, 512-bit bus, 575 W). https://www.nvidia.com/fr-fr/geforce/graphics-cards/50-series/rtx-5090/ ↩ ↩2
-
NVIDIA, NVIDIA AI Enterprise — End-of-Life Notices (GPUs retired from Infra 8.0 onward: Tesla V100, RTX 4000 SFF Ada, RTX A4000, Quadro RTX; A100, H100/H200, L40/L40S, and RTX 6000 Ada still supported), re-read on 2026-10-10. https://docs.nvidia.com/ai-enterprise/lifecycle/latest/eol-notices.html ↩
-
vLLM, Parallelism and Scaling (tensor parallel, pipeline parallel, multi-GPU requirements). https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ ↩
-
NVIDIA, TensorRT-LLM and datacenter GPU documentation. https://nvidia.github.io/TensorRT-LLM/ ↩
-
PyPI, tensorrt-llm (1.3.0 stable of 2026-11-20 after release candidates rc0 → rc29; 1.4.0 in release candidates since 2026-10-13), captured 2026-11-20. https://pypi.org/project/tensorrt-llm/ · vLLM Project, Release v0.28.0 (CUDA 13 by default), August 2026. https://github.com/vllm-project/vllm/releases/tag/v0.28.0 · SGLang Project, Release v0.5.20 (CUDA 13 required), 18 September 2026. https://github.com/sgl-project/sglang/releases/tag/v0.5.20 ↩