Skip to content

🏎️ Memory Bandwidth & The "Memory Wall"

You may have heard: “To run an LLM, you need a powerful GPU.” That’s true, but incomplete. In practice, the fastest card on the market can saturate and become as slow as an entry-level card — if memory cannot keep up.

That’s the paradox this chapter explains. Understanding memory bandwidth means understanding why your setup generates 4 tokens/s instead of 40 — and how to fix it.

In system architecture applied to AI, one reality keeps coming back: for LLM inference, the limit is often memory before raw compute1.

This phenomenon is classically called the “Memory Wall”. During autoregressive generation, the GPU/CPU alternates very fast compute phases and phases waiting for data from memory. Final throughput is therefore strongly correlated with memory bandwidth (GB/s), not just TFLOPS1.


🧠 Inference physics: Prefill vs Decoding

To understand the bottleneck, separate two phases:

graph TD
    A[Prompt input] --> B(Phase 1: Prefill)
    B -->|Heavy parallel compute| C[Limited by TFLOPS]
    C --> D(Phase 2: Decoding / Generation)
    D -->|Sequential memory reads| E[Limited by bandwidth]

1. The “Prefill” phase (Context ingestion)

The model processes the input prompt in parallel (large matrices).

  • Hardware behavior: better use of compute units.
  • Dominant factor: compute + memory mix, often more favorable to compute than decoding.

2. The “Decoding” phase (Word-by-word generation)

The model generates one token, then repeats for the next. This process is sequential.

  • Hardware behavior: repeated reads of weights + KV Cache management; arithmetic intensity is lower than in prefill.
  • Dominant factor: memory bandwidth, especially at low batch size1.

📐 The throughput equation

For quick sizing, use an upper bound:

Max speed (tokens/s)=Memory bandwidth (GB/s)Model size in memory (GB)\text{Max speed (tokens/s)} = \frac{\text{Memory bandwidth (GB/s)}}{\text{Model size in memory (GB)}}

Practical case: Dense 70B model in 4-bit quantization (Q4)

A dense 70B model quantized to 4-bit uses about 40 GB of weight memory (order of magnitude).

  1. On a classic PC (DDR5 dual channel):
    • Real bandwidth: ∼100 GB/s\sim 100 \text{ GB/s}
    • Calculation: 100 GB/s40 GB=2.5 tokens/s\frac{100 \text{ GB/s}}{40 \text{ GB}} = \mathbf{2.5 \text{ tokens/s}} (theoretical bound).
  2. On AMD Ryzen AI Max+ PRO 495 (Gorgon Halo, e.g. Framework Desktop 192 GB)34:
    • Real bandwidth: ∼273 GB/s\sim 273 \text{ GB/s}
    • Calculation: 273 GB/s40 GB=6.8 tokens/s\frac{273 \text{ GB/s}}{40 \text{ GB}} = \mathbf{6.8 \text{ tokens/s}} (theoretical bound).
  3. On Mac Studio M5 Max (high-end unified memory):
    • Bandwidth: 614 GB/s614 \text{ GB/s} (announced by Apple in August 2026)5
    • Calculation: 614 GB/s40 GB=15.4 tokens/s\frac{614 \text{ GB/s}}{40 \text{ GB}} = \mathbf{15.4 \text{ tokens/s}} (theoretical bound); an M5 Ultra (1,200 GB/s1{,}200 \text{ GB/s}) doubles this bound to 30 tokens/s\mathbf{30 \text{ tokens/s}}5.
  4. On Nvidia RTX 5090 (dedicated GDDR7 VRAM — Blackwell):
    • Real bandwidth: 1,792 GB/s1{,}792 \text{ GB/s}
    • Calculation: 1792 GB/s40 GB=44.8 tokens/s\frac{1792 \text{ GB/s}}{40 \text{ GB}} = \mathbf{44.8 \text{ tokens/s}} (theoretical bound).
    • Bandwidth example only: in Q3 2026 the RTX 5090 sells for ≥ $5,000 (see Stations Multi-GPU) — the RTX PRO 6000 Blackwell offers the same bandwidth (1,792 GB/s) with 96 GB 6.

📊 Storage technology comparison

Values below are order-of-magnitude guides for architecture (real performance varies by software stack and workload).

TechnologyBandwidth (order of magnitude)SourceImpact on inference
10 GbE Ethernet∼1.25 GB/s\sim 1.25 \text{ GB/s}10 Gbit/s conversiontoo low to “extend” a model online without heavy penalty
PCIe 5.0 x16∼64 GB/s\sim 64 \text{ GB/s} (per direction)bus specbecomes a bottleneck for frequent CPU↔GPU transfers
DDR5 desktop RAM∼80\sim 80 to 100 GB/s100 \text{ GB/s}typical dual-channel platformshigh capacity, limited throughput for large LLMs
AMD Ryzen AI Max PRO 400 unified memoryup to ∼273 GB/s\sim 273 \text{ GB/s}3interesting capacity/bandwidth trade-off on x86
NVIDIA DGX Spark unified memory (LPDDR5x)∼273 GB/s\sim 273 \text{ GB/s}7same class as the AMD APU; native CUDA and FP4
Apple M5 Max / M5 Ultra unified memory614614 to 1,200 GB/s1{,}200 \text{ GB/s}5excellent local throughput without PCIe offload
RTX 5090 VRAM (GDDR7)∼1.79 TB/s\sim 1.79 \text{ TB/s}89very high throughput for fast decoding

Two dated remarks (Q4 2026). Intel is preparing a “capacity-first” inference GPU (Crescent Island, see 🧩 Stations Multi-GPU) whose bandwidth is not published: the formula above already says it will be limited in single-request decode and relevant in batch. And RAM is no longer the cheap option in the table: DRAM is still rising every quarter (see 💰 Comparaison TCO), so the “capacity vs bandwidth” trade-off is now also made in euros per GB.


Network clustering limits


📚 Sources and references

Footnotes

  1. Amir Gholami et al., AI and Memory Wall (arXiv:2403.14123), 2024. https://arxiv.org/abs/2403.14123 ↩ ↩2 ↩3

  2. Meta, Llama 4 Model Card (Scout/Maverick, MoE, no dense “70B” variant), 2025. https://raw.githubusercontent.com/meta-llama/llama-models/main/models/llama4/MODEL_CARD.md ↩

  3. ServeTheHome, AMD Ups Ante With 192GB Ryzen AI Max PRO 400 Chips for AI Systems, 2026. https://www.servethehome.com/amd-reveals-ryzen-ai-max-pro-400-series-192gb-ram-for-ai-systems/ ↩ ↩2

  4. AMD, AMD Ryzen AI Max+ PRO 495 — product page (192 GB LPDDR5X-8533, 256-bit bus), consulted 2026-10-09. https://www.amd.com/en/products/processors/laptop/ryzen-pro/ai-max-pro-400-series/amd-ryzen-ai-max-plus-pro-495.html · Framework, The 192GB Framework Desktop is open for pre-order (pre-orders from 2026-09-30, shipping November 2026), 30 September 2026. https://frame.work/blog/192gb-framework-desktop-open-for-pre-order ↩

  5. Apple, Mac Studio — Technical Specifications (M5 Max 614 GB/s, M5 Ultra 1.2 TB/s; the page has described the M5 generation since 2026-08-25, the M4 Max 546 GB/s / M3 Ultra 819 GB/s values remain on the “Mac Studio (2025)” sheet), re-read on 2026-10-09. https://www.apple.com/mac-studio/specs/ · Apple Support, Mac Studio (2025) — Tech Specs. https://support.apple.com/en-us/122211 ↩ ↩2 ↩3

  6. Tom’s Hardware, Nvidia’s RTX 5090 vanishes from online retail in the US — third-party sellers now demand as much as $9,500 (MSRP $1,999), 14 September 2026. https://www.tomshardware.com/pc-components/gpus/nvidias-rtx-5090-vanishes-from-online-retail-in-the-us-third-party-sellers-now-demand-as-much-as-usd9-500-for-nvidias-fastest-gpu · NVIDIA, RTX PRO 6000 Blackwell Workstation Edition (96 GB GDDR7 ECC, 1,792 GB/s). https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000/ ↩

  7. NVIDIA, DGX Spark — product page (Grace Blackwell GB10, 128 GB LPDDR5x, ~273 GB/s, native FP4), re-read on 2026-10-09. https://www.nvidia.com/en-us/products/workstations/dgx-spark/ ↩

  8. NVIDIA, GeForce RTX 5090 product page (32 GB GDDR7, 512-bit bus, TGP 575 W). https://www.nvidia.com/fr-fr/geforce/graphics-cards/50-series/rtx-5090/ ↩

  9. TechPowerUp, NVIDIA GeForce RTX 5090 Specs (memory bandwidth 1.79 TB/s), 2026. https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216 ↩

  10. Apple MLX, Distributed Communication (“Starting from macOS 26.2, RDMA over thunderbolt is available”; JACCL backend), consulted 2026-10-10. https://ml-explore.github.io/mlx/build/html/usage/distributed.html ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.