Q4 Quantization
📝 Short definition
Quantization where weights are compressed to about 4 bits per parameter, dividing memory footprint by ~4 compared to native BF16. The Q4_K_M format is the most used in the Ollama / llama.cpp ecosystem.
📖 Detailed definition
Q4_K_M means:
- Q4: 4 bits per parameter (~0.5 effective byte with block metadata)
- K: the “K-quants” family from llama.cpp. Weights are quantized in blocks of 32 values, grouped into super-blocks of 256, with scales that are themselves quantized. It is not K-means clustering.
- M: the “medium” variant of the tensor mix. Q4_K_M keeps some of the most sensitive tensors (half of
attention.wvandfeed_forward.w2) in Q6_K and puts the rest in Q4_K. The S, M and L suffixes denote this mix, not a block size. It is the quality/speed compromise recommended for most uses.
Typical VRAM footprints in Q4_K_M:
| Model | Q4_K_M | BF16 (reference) | Gain |
|---|---|---|---|
| 7B / 8B | ~5 GB | ~14 GB | ×2.8 |
| 14B | ~9 GB | ~28 GB | ×3.1 |
| 32B | ~20 GB | ~64 GB | ×3.2 |
| 70B | ~40 GB | ~140 GB | ×3.5 |
💡 Why it matters for on-prem AI
Q4_K_M lets a user with a ~€3,500 machine (16–24 GB VRAM) run a 70B model with offloading, or load a 70B comfortably on a 128 GB APU with headroom for KV Cache.
It is also Ollama’s default download format: ollama pull llama3.1:70b automatically downloads a Q4_K_M build.
⚠️ Common pitfalls
- Q4 is not suited to every task: for sensitive uses (precise code editing, medical or legal extraction, custodian agents), overly aggressive quantization can miss nuances or produce partially correct outputs. Prefer Q6_K or Q8 if VRAM allows.
- Do not confuse with Q4_0 (simpler, lower quality than Q4_K_M) or Q4_K_S (lighter, slightly less accurate than K_M).
- Stated footprint is weights only: KV Cache adds on top depending on context length.
📚 Go deeper
- 🗜️ 4-bit & 8-bit Quantization — full mathematical mechanism
- 🛠️ Scenario A — concrete Q4 usage with offloading
🔗 See also
- Quantization — overview (Q8, AWQ, FP8)
- GGUF — file format that embeds quantization
- Offloading
Reducing numerical precision of LLM weights to lower memory footprint and speed up inference.Technique that places part of the model in RAM or on SSD when VRAM is insufficient, at the cost of reduced throughput.Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.Portable file format for local inference with llama.cpp, optimized for K-quant quantization.