Offloading
π Short definition
Technique that moves part of a modelβs weights out of VRAM (into system RAM or on SSD) so a model too large for VRAM can still run, accepting reduced generation throughput.
π Detailed definition
When a 40 GB model does not fit in 24 GB of VRAM, llama.cpp can load about twenty GB into VRAM and keep the rest in RAM. During inference, the βoff VRAMβ layers are computed by the CPU directly in RAM (-ngl option): only the activations cross PCIe.
The bottleneck is system RAM bandwidth (~80β100 GB/s with dual-channel DDR5); PCIe only becomes limiting if weights are copied to the GPU on every pass:
| Connection | Effective bandwidth | Impact on tokens/s |
|---|---|---|
| PCIe 5.0 x16 | ~64 GB/s per direction (~128 GB/s bidirectional) | Limiting if weights are copied on every pass |
| PCIe 4.0 x16 | ~32 GB/s per direction (~64 GB/s bidirectional) | Same, twice as slow |
| Thunderbolt 4 (external GPU) | ~4 GB/s (32 Gb/s PCIe tunnel) | Prohibitive if weights are copied on every pass; otherwise, only slows down initial loading |
Approximate throughput formula for partial offloading (P_VRAM and P_RAM: GB of weights placed in VRAM and in RAM; D: bandwidth in GB/s):
π‘ Why it matters for on-prem AI
Offloading is at the heart of Scenario A (Dev Lab): a machine under β¬3,500 with 16β24 GB VRAM can run a 70B model in Q4 (~40 GB) by offloading excess layers to DDR5. Throughput will cap at about 4β5 tokens/s (theoretical bound; measured throughput is lower, see Scenario A) rather than 15β40 tokens/s, but that is enough for development and testing.
β οΈ Common pitfalls
- Offloading and concurrency do not mix well: with 5 simultaneous users, each request rereads the layers left in RAM, which are already bounded by DDR5 bandwidth, collapsing global throughput.
- Offloading to SSD (instead of RAM) is possible but ~10Γ slower β reserve it for initial loading, not real-time inference.
- Assuming more RAM always compensates: capacity lets you load the model, but throughput remains bounded by RAM bandwidth (~80β100 GB/s with dual-channel DDR5), not by its size.