Quantization
π Short definition
Technique that reduces the numerical precision of model weights (e.g. FP16 β INT8 β Q4) to lower VRAM footprint and, in some cases, speed up inference.
π Detailed definition
An LLM stores its parameters as floating-point values. Native training precision is BF16 or FP16 (2 bytes per parameter). Quantization compresses those values toward less precise formats:
| Format | Bytes/parameter | 70B footprint | Quality loss |
|---|---|---|---|
| BF16 / FP16 | 2.0 | ~140 GB | Reference |
| INT8 / Q8 | 1.0 | ~70 GB | Very low |
| Q4_K_M | ~0.5 | ~40 GB | Low to moderate |
| Q2_K | ~0.3β0.4 | ~25 GB | Significant1 |
Three main families of methods:
- GGUF / llama.cpp: portable format for Ollama and workstations. Includes Q2 through Q8 variants; the βKβ variants (K-quants) split weights into 256-value super-blocks with several scales and keep sensitive tensors at higher precision, which preserves quality better at equal size2.
- AWQ / GPTQ: 4-bit quantizations for GPUs (AWQ protects ~1% of salient weights based on activations; GPTQ compensates rounding using the Hessian), optimized for vLLM/SGLang with dedicated kernels. Quality comparable to K-quants at equal bits, higher GPU throughput3.
- FP8 / FP4: low-resolution floating precisions. FP8 is native on Ada Lovelace (L40S, RTX 6000 Ada, RTX 40), Hopper (H100, H200) and Blackwell; FP4 (NVFP4, MXFP4) is native on Blackwell only (B200, RTX 50, RTX PRO 6000, DGX Spark). Since 2026, vendors ship FP4 weights directly (gpt-oss and Kimi K3 in MXFP4; Nemotron 3.5 Lightning in NVFP4, β 22 GB), including for workstation GPUs4.
Fourth trend in 2026: quantization-aware training (QAT). Kimi K3 is trained in MXFP4 from the SFT phase onward, and NVIDIA publishes Nemotron 3.5 Lightning in NVFP4 as the βrecommended path for deploymentβ, with BF16 weights reserved for post-training. The vendorβs 4-bit checkpoint then becomes the canonical version, and re-quantizing it yourself brings nothing4.
π‘ Why it matters for on-prem AI
Quantization is the number one lever for fitting a large model on your hardware. A 70B model inaccessible in BF16 (140 GB) becomes usable in Q4_K_M (~40 GB) on an APU with 128 GB of unified memory.
β οΈ Common pitfalls
- Confusing weights footprint (fixed) with KV Cache footprint (dynamic, depends on context). A Q4 model can still OOM if context is long.
- Assuming Q4 is always enough: on critical tasks (code editing, medical extraction), Q4 can measurably degrade reliability. Test with your golden dataset.
- Comparing benchmark scores between a BF16 model and a Q4 model as if they were identical.
π Go deeper
- ποΈ 4-bit & 8-bit Quantization β math mechanism, formula, perplexity/VRAM trade-off
π See also
- Q4_K_M Quantization β the most common practical format, uses and limits
- VRAM
- GGUF
Footnotes
-
unsloth, DeepSeek-V3-GGUF (Q2_K_XS β 221 GB for 671 B, i.e. β 2.6 bits/weight; Q4_K_M β 404 GB), January 2025; J. Wang et al., arXiv:2601.14277 (effective bits of K-quants), January 2026. https://huggingface.co/unsloth/DeepSeek-V3-GGUF Β· https://arxiv.org/abs/2601.14277 β©
-
J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv:2601.14277; description of K-quants by super-blocks), January 2026. https://arxiv.org/abs/2601.14277 β©
-
J. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978), 2023; E. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323), 2022. https://arxiv.org/abs/2306.00978 Β· https://arxiv.org/abs/2210.17323 β©
-
OpenAI, gpt-oss-120b (βpost-trained with MXFP4 quantization of the MoE weightsβ, 20b in 16 GB, Apache 2.0), August 2025. https://huggingface.co/openai/gpt-oss-120b; Moonshot AI, Kimi K3 (βMXFP4 weights / MXFP8 activations (quantization-aware training)β from SFT onward), July 2026. https://huggingface.co/moonshotai/Kimi-K3; NVIDIA, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (β 21.6 GB of safetensors; βthe NVFP4 release is the recommended pathβ for deployment), August 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 β© β©2