🖥️ Index — Hardware
Hardware dictates what you can load in VRAM, what memory throughput you can reach during decoding, and whether you can run tensor parallelism or offloading.
Path by scenario
| Blueprint | Priority hardware pages |
|---|---|
| A — Dev Lab | APU & unified memory, Multi-GPU workstations (offloading) |
| B — SMB Appliance | APU & unified memory, GPU rack servers (1–2 GPU) |
| C — Office cluster | Multi-GPU workstations, AI networking (Thunderbolt 5 + RDMA) |
| D — Datacenter | GPU rack servers, RoCE / InfiniBand |
Chapter pages
- 🧠 APU & Unified Memory — Apple M5, AMD Ryzen AI Max PRO 400, NVIDIA DGX Spark 64/128 GB
- 🧩 Multi-GPU Workstations — PCIe workstations (no NVLink since Ada), PCIe switches, L40S / A100 servers
- 🏭 GPU rack servers — 1U/2U/4U, HGX, RTX vs datacenter choice
- 🌐 AI networking — RoCE, InfiniBand, Thunderbolt
Useful hardware lexicon
VRAM · HBM · NVLink · PCIe · Multi-GPU · Unified memory
Very high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.Technique that places part of the model in RAM or on SSD when VRAM is insufficient, at the cost of reduced throughput.Stacked memory with very high bandwidth, used on professional AI accelerators.Dedicated hardware link that connects NVIDIA GPUs at very high speed, bypassing the PCIe bus; it is gone from recent workstation and consumer cards.Standard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink.Architecture using several GPUs in one machine or a cluster to increase memory and throughput; the gains depend on the interconnect (PCIe, NVLink).Architecture where CPU, GPU and sometimes NPU share a single memory pool, avoiding some copies over PCIe; found on Apple Silicon and some recent AMD APUs.