Skip to content

Offloading

πŸ“ Short definition

Technique that moves part of a model’s weights out of VRAM (into system RAM or on SSD) so a model too large for VRAM can still run, accepting reduced generation throughput.

πŸ“– Detailed definition

When a 40 GB model does not fit in 24 GB of VRAM, llama.cpp can load about twenty GB into VRAM and keep the rest in RAM. During inference, the β€œoff VRAM” layers are computed by the CPU directly in RAM (-ngl option): only the activations cross PCIe.

The bottleneck is system RAM bandwidth (~80–100 GB/s with dual-channel DDR5); PCIe only becomes limiting if weights are copied to the GPU on every pass:

ConnectionEffective bandwidthImpact on tokens/s
PCIe 5.0 x16~64 GB/s per direction (~128 GB/s bidirectional)Limiting if weights are copied on every pass
PCIe 4.0 x16~32 GB/s per direction (~64 GB/s bidirectional)Same, twice as slow
Thunderbolt 4 (external GPU)~4 GB/s (32 Gb/s PCIe tunnel)Prohibitive if weights are copied on every pass; otherwise, only slows down initial loading

Approximate throughput formula for partial offloading (P_VRAM and P_RAM: GB of weights placed in VRAM and in RAM; D: bandwidth in GB/s):

tokens/sβ‰ˆ1PVRAMDVRAM+PRAMDRAM\text{tokens/s} \approx \frac{1}{\frac{P_{VRAM}}{D_{VRAM}} + \frac{P_{RAM}}{D_{RAM}}}

πŸ’‘ Why it matters for on-prem AI

Offloading is at the heart of Scenario A (Dev Lab): a machine under €3,500 with 16–24 GB VRAM can run a 70B model in Q4 (~40 GB) by offloading excess layers to DDR5. Throughput will cap at about 4–5 tokens/s (theoretical bound; measured throughput is lower, see Scenario A) rather than 15–40 tokens/s, but that is enough for development and testing.

⚠️ Common pitfalls

  • Offloading and concurrency do not mix well: with 5 simultaneous users, each request rereads the layers left in RAM, which are already bounded by DDR5 bandwidth, collapsing global throughput.
  • Offloading to SSD (instead of RAM) is possible but ~10Γ— slower β€” reserve it for initial loading, not real-time inference.
  • Assuming more RAM always compensates: capacity lets you load the model, but throughput remains bounded by RAM bandwidth (~80–100 GB/s with dual-channel DDR5), not by its size.

πŸ“š Go deeper

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.