RAM
π Short definition
The machineβs primary memory, used by the CPU for the OS and applications. In local AI, it serves as overflow when VRAM is insufficient to load an entire model.
π Detailed definition
A DDR5-5600 stick in dual-channel offers about 85β90 GB/s bandwidth under ideal conditions. That is roughly 15Γ less than GDDR6X VRAM (1,000+ GB/s) and 30Γ less than datacenter HBM3.
In LLM inference, this difference translates directly into generation throughput: a 7B model entirely in VRAM generates 40β60 tokens/s; the same model partly in RAM often falls to 5β15 tokens/s, because the layers left in RAM are computed by the CPU at DDR5 speed (~85β90 GB/s), with only the activations crossing PCIe.
RAM remains useful in three cases:
| Case | Usage |
|---|---|
| CPU offloading (llama.cpp) | Part of the model in RAM, part in VRAM β capacity/throughput trade-off |
| Unified memory (Apple/AMD APU) | RAM is VRAM β no PCIe copy |
| Preloading | The GGUF file is read from SSD and paged into RAM cache before transfer to GPU |
π‘ Why it matters for on-prem AI
Scenario A (Dev Lab) explicitly relies on RAM offloading to run a 70B model on a machine costing under β¬3,500 equipped with a single 24 GB GPU: the layers that do not fit in VRAM are computed by the CPU in RAM. It is the accessible entry point for a solo developer.
β οΈ Common pitfalls
- Believing 128 GB of RAM replaces 128 GB of VRAM: throughput is 10β30Γ lower.
- Ignoring page cache: reading a 140 GB GGUF from SSD fills available RAM before inference even starts, risking OOM on tight machines.
- Forgetting that offloading is catastrophic under concurrency: each extra user multiplies the reads of the layers left in RAM, which are already bounded by DDR5 bandwidth.