🏎️ Memory Bandwidth & The "Memory Wall"
You may have heard: “To run an LLM, you need a powerful GPU.” That’s true, but incomplete. In practice, the fastest card on the market can saturate and become as slow as an entry-level card — if memory cannot keep up.
That’s the paradox this chapter explains. Understanding memory bandwidth means understanding why your setup generates 4 tokens/s instead of 40 — and how to fix it.
In system architecture applied to AI, one reality keeps coming back: for LLM inference, the limit is often memory before raw compute1.
This phenomenon is classically called the “Memory Wall”. During autoregressive generation, the GPU/CPU alternates very fast compute phases and phases waiting for data from memory. Final throughput is therefore strongly correlated with memory bandwidth (GB/s), not just TFLOPS1.
🧠 Inference physics: Prefill vs Decoding
To understand the bottleneck, separate two phases:
graph TD
A[Prompt input] --> B(Phase 1: Prefill)
B -->|Heavy parallel compute| C[Limited by TFLOPS]
C --> D(Phase 2: Decoding / Generation)
D -->|Sequential memory reads| E[Limited by bandwidth]
1. The “Prefill” phase (Context ingestion)
The model processes the input prompt in parallel (large matrices).
- Hardware behavior: better use of compute units.
- Dominant factor: compute + memory mix, often more favorable to compute than decoding.
2. The “Decoding” phase (Word-by-word generation)
The model generates one token, then repeats for the next. This process is sequential.
- Hardware behavior: repeated reads of weights + KV Cache management; arithmetic intensity is lower than in prefill.
- Dominant factor: memory bandwidth, especially at low batch size1.
📐 The throughput equation
For quick sizing, use an upper bound:
Practical case: Dense 70B model in 4-bit quantization (Q4)
A dense 70B model quantized to 4-bit uses about 40 GB of weight memory (order of magnitude).
- On a classic PC (DDR5 dual channel):
- Real bandwidth:
- Calculation: (theoretical bound).
- On AMD Ryzen AI Max+ PRO 495 (Gorgon Halo, e.g. Framework Desktop 192 GB)34:
- Real bandwidth:
- Calculation: (theoretical bound).
- On Mac Studio M5 Max (high-end unified memory):
- On Nvidia RTX 5090 (dedicated GDDR7 VRAM — Blackwell):
- Real bandwidth:
- Calculation: (theoretical bound).
- Bandwidth example only: in Q3 2026 the RTX 5090 sells for ≥ $5,000 (see Stations Multi-GPU) — the RTX PRO 6000 Blackwell offers the same bandwidth (1,792 GB/s) with 96 GB 6.
📊 Storage technology comparison
Values below are order-of-magnitude guides for architecture (real performance varies by software stack and workload).
| Technology | Bandwidth (order of magnitude) | Source | Impact on inference |
|---|---|---|---|
| 10 GbE Ethernet | 10 Gbit/s conversion | too low to “extend” a model online without heavy penalty | |
| PCIe 5.0 x16 | (per direction) | bus spec | becomes a bottleneck for frequent CPU↔GPU transfers |
| DDR5 desktop RAM | to | typical dual-channel platforms | high capacity, limited throughput for large LLMs |
| AMD Ryzen AI Max PRO 400 unified memory | up to | 3 | interesting capacity/bandwidth trade-off on x86 |
| NVIDIA DGX Spark unified memory (LPDDR5x) | 7 | same class as the AMD APU; native CUDA and FP4 | |
| Apple M5 Max / M5 Ultra unified memory | to | 5 | excellent local throughput without PCIe offload |
| RTX 5090 VRAM (GDDR7) | 89 | very high throughput for fast decoding |
Two dated remarks (Q4 2026). Intel is preparing a “capacity-first” inference GPU (Crescent Island, see 🧩 Stations Multi-GPU) whose bandwidth is not published: the formula above already says it will be limited in single-request decode and relevant in batch. And RAM is no longer the cheap option in the table: DRAM is still rising every quarter (see 💰 Comparaison TCO), so the “capacity vs bandwidth” trade-off is now also made in euros per GB.
Network clustering limits
📚 Sources and references
Footnotes
-
Amir Gholami et al., AI and Memory Wall (arXiv:2403.14123), 2024. https://arxiv.org/abs/2403.14123 ↩ ↩2 ↩3
-
Meta, Llama 4 Model Card (Scout/Maverick, MoE, no dense “70B” variant), 2025. https://raw.githubusercontent.com/meta-llama/llama-models/main/models/llama4/MODEL_CARD.md ↩
-
ServeTheHome, AMD Ups Ante With 192GB Ryzen AI Max PRO 400 Chips for AI Systems, 2026. https://www.servethehome.com/amd-reveals-ryzen-ai-max-pro-400-series-192gb-ram-for-ai-systems/ ↩ ↩2
-
AMD, AMD Ryzen AI Max+ PRO 495 — product page (192 GB LPDDR5X-8533, 256-bit bus), consulted 2026-10-09. https://www.amd.com/en/products/processors/laptop/ryzen-pro/ai-max-pro-400-series/amd-ryzen-ai-max-plus-pro-495.html · Framework, The 192GB Framework Desktop is open for pre-order (pre-orders from 2026-09-30, shipping November 2026), 30 September 2026. https://frame.work/blog/192gb-framework-desktop-open-for-pre-order ↩
-
Apple, Mac Studio — Technical Specifications (M5 Max 614 GB/s, M5 Ultra 1.2 TB/s; the page has described the M5 generation since 2026-08-25, the M4 Max 546 GB/s / M3 Ultra 819 GB/s values remain on the “Mac Studio (2025)” sheet), re-read on 2026-10-09. https://www.apple.com/mac-studio/specs/ · Apple Support, Mac Studio (2025) — Tech Specs. https://support.apple.com/en-us/122211 ↩ ↩2 ↩3
-
Tom’s Hardware, Nvidia’s RTX 5090 vanishes from online retail in the US — third-party sellers now demand as much as $9,500 (MSRP $1,999), 14 September 2026. https://www.tomshardware.com/pc-components/gpus/nvidias-rtx-5090-vanishes-from-online-retail-in-the-us-third-party-sellers-now-demand-as-much-as-usd9-500-for-nvidias-fastest-gpu · NVIDIA, RTX PRO 6000 Blackwell Workstation Edition (96 GB GDDR7 ECC, 1,792 GB/s). https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000/ ↩
-
NVIDIA, DGX Spark — product page (Grace Blackwell GB10, 128 GB LPDDR5x, ~273 GB/s, native FP4), re-read on 2026-10-09. https://www.nvidia.com/en-us/products/workstations/dgx-spark/ ↩
-
NVIDIA, GeForce RTX 5090 product page (32 GB GDDR7, 512-bit bus, TGP 575 W). https://www.nvidia.com/fr-fr/geforce/graphics-cards/50-series/rtx-5090/ ↩
-
TechPowerUp, NVIDIA GeForce RTX 5090 Specs (memory bandwidth 1.79 TB/s), 2026. https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216 ↩
-
Apple MLX, Distributed Communication (“Starting from macOS 26.2, RDMA over thunderbolt is available”; JACCL backend), consulted 2026-10-10. https://ml-explore.github.io/mlx/build/html/usage/distributed.html ↩
Incoming links
- 🏢 Scenario B: SME Appliance (Unified Memory)
- 🧠 The Journey of a Prompt: How Does an LLM Work?
- 🚀 Getting started with Ollama
- 🚀 Zero to Hero Index
- 🛠️ Scenario A: Dev Lab (GPU PC or unified memory)
- Decoding
- HBM
- KV Cache
- LLM inference
- LPDDR5X
- Memory bandwidth
- Memory Wall
- NPU
- Prefill
- RDMA
- TFLOPS
- Zero to Hero: On-Premise AI