Skip to content

RAM

πŸ“ Short definition

The machine’s primary memory, used by the CPU for the OS and applications. In local AI, it serves as overflow when VRAM is insufficient to load an entire model.

πŸ“– Detailed definition

A DDR5-5600 stick in dual-channel offers about 85–90 GB/s bandwidth under ideal conditions. That is roughly 15Γ— less than GDDR6X VRAM (1,000+ GB/s) and 30Γ— less than datacenter HBM3.

In LLM inference, this difference translates directly into generation throughput: a 7B model entirely in VRAM generates 40–60 tokens/s; the same model partly in RAM often falls to 5–15 tokens/s, because the layers left in RAM are computed by the CPU at DDR5 speed (~85–90 GB/s), with only the activations crossing PCIe.

RAM remains useful in three cases:

CaseUsage
CPU offloading (llama.cpp)Part of the model in RAM, part in VRAM β€” capacity/throughput trade-off
Unified memory (Apple/AMD APU)RAM is VRAM β€” no PCIe copy
PreloadingThe GGUF file is read from SSD and paged into RAM cache before transfer to GPU

πŸ’‘ Why it matters for on-prem AI

Scenario A (Dev Lab) explicitly relies on RAM offloading to run a 70B model on a machine costing under €3,500 equipped with a single 24 GB GPU: the layers that do not fit in VRAM are computed by the CPU in RAM. It is the accessible entry point for a solo developer.

⚠️ Common pitfalls

  • Believing 128 GB of RAM replaces 128 GB of VRAM: throughput is 10–30Γ— lower.
  • Ignoring page cache: reading a 140 GB GGUF from SSD fills available RAM before inference even starts, risking OOM on tight machines.
  • Forgetting that offloading is catastrophic under concurrency: each extra user multiplies the reads of the layers left in RAM, which are already bounded by DDR5 bandwidth.

πŸ“š Go deeper

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.