Skip to content

Q4 Quantization

📝 Short definition

Quantization where weights are compressed to about 4 bits per parameter, dividing memory footprint by ~4 compared to native BF16. The Q4_K_M format is the most used in the Ollama / llama.cpp ecosystem.

📖 Detailed definition

Q4_K_M means:

  • Q4: 4 bits per parameter (~0.5 effective byte with block metadata)
  • K: the “K-quants” family from llama.cpp. Weights are quantized in blocks of 32 values, grouped into super-blocks of 256, with scales that are themselves quantized. It is not K-means clustering.
  • M: the “medium” variant of the tensor mix. Q4_K_M keeps some of the most sensitive tensors (half of attention.wv and feed_forward.w2) in Q6_K and puts the rest in Q4_K. The S, M and L suffixes denote this mix, not a block size. It is the quality/speed compromise recommended for most uses.

Typical VRAM footprints in Q4_K_M:

ModelQ4_K_MBF16 (reference)Gain
7B / 8B~5 GB~14 GB×2.8
14B~9 GB~28 GB×3.1
32B~20 GB~64 GB×3.2
70B~40 GB~140 GB×3.5

💡 Why it matters for on-prem AI

Q4_K_M lets a user with a ~€3,500 machine (16–24 GB VRAM) run a 70B model with offloading, or load a 70B comfortably on a 128 GB APU with headroom for KV Cache.

It is also Ollama’s default download format: ollama pull llama3.1:70b automatically downloads a Q4_K_M build.

⚠️ Common pitfalls

  • Q4 is not suited to every task: for sensitive uses (precise code editing, medical or legal extraction, custodian agents), overly aggressive quantization can miss nuances or produce partially correct outputs. Prefer Q6_K or Q8 if VRAM allows.
  • Do not confuse with Q4_0 (simpler, lower quality than Q4_K_M) or Q4_K_S (lighter, slightly less accurate than K_M).
  • Stated footprint is weights only: KV Cache adds on top depending on context length.

📚 Go deeper

🔗 See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.