Skip to content

Quantization

πŸ“ Short definition

Technique that reduces the numerical precision of model weights (e.g. FP16 β†’ INT8 β†’ Q4) to lower VRAM footprint and, in some cases, speed up inference.

πŸ“– Detailed definition

An LLM stores its parameters as floating-point values. Native training precision is BF16 or FP16 (2 bytes per parameter). Quantization compresses those values toward less precise formats:

FormatBytes/parameter70B footprintQuality loss
BF16 / FP162.0~140 GBReference
INT8 / Q81.0~70 GBVery low
Q4_K_M~0.5~40 GBLow to moderate
Q2_K~0.3–0.4~25 GBSignificant1

Three main families of methods:

  • GGUF / llama.cpp: portable format for Ollama and workstations. Includes Q2 through Q8 variants; the β€œK” variants (K-quants) split weights into 256-value super-blocks with several scales and keep sensitive tensors at higher precision, which preserves quality better at equal size2.
  • AWQ / GPTQ: 4-bit quantizations for GPUs (AWQ protects ~1% of salient weights based on activations; GPTQ compensates rounding using the Hessian), optimized for vLLM/SGLang with dedicated kernels. Quality comparable to K-quants at equal bits, higher GPU throughput3.
  • FP8 / FP4: low-resolution floating precisions. FP8 is native on Ada Lovelace (L40S, RTX 6000 Ada, RTX 40), Hopper (H100, H200) and Blackwell; FP4 (NVFP4, MXFP4) is native on Blackwell only (B200, RTX 50, RTX PRO 6000, DGX Spark). Since 2026, vendors ship FP4 weights directly (gpt-oss and Kimi K3 in MXFP4; Nemotron 3.5 Lightning in NVFP4, β‰ˆ 22 GB), including for workstation GPUs4.

Fourth trend in 2026: quantization-aware training (QAT). Kimi K3 is trained in MXFP4 from the SFT phase onward, and NVIDIA publishes Nemotron 3.5 Lightning in NVFP4 as the β€œrecommended path for deployment”, with BF16 weights reserved for post-training. The vendor’s 4-bit checkpoint then becomes the canonical version, and re-quantizing it yourself brings nothing4.

πŸ’‘ Why it matters for on-prem AI

Quantization is the number one lever for fitting a large model on your hardware. A 70B model inaccessible in BF16 (140 GB) becomes usable in Q4_K_M (~40 GB) on an APU with 128 GB of unified memory.

⚠️ Common pitfalls

  • Confusing weights footprint (fixed) with KV Cache footprint (dynamic, depends on context). A Q4 model can still OOM if context is long.
  • Assuming Q4 is always enough: on critical tasks (code editing, medical extraction), Q4 can measurably degrade reliability. Test with your golden dataset.
  • Comparing benchmark scores between a BF16 model and a Q4 model as if they were identical.

πŸ“š Go deeper

πŸ”— See also

Footnotes

  1. unsloth, DeepSeek-V3-GGUF (Q2_K_XS β‰ˆ 221 GB for 671 B, i.e. β‰ˆ 2.6 bits/weight; Q4_K_M β‰ˆ 404 GB), January 2025; J. Wang et al., arXiv:2601.14277 (effective bits of K-quants), January 2026. https://huggingface.co/unsloth/DeepSeek-V3-GGUF Β· https://arxiv.org/abs/2601.14277 ↩

  2. J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv:2601.14277; description of K-quants by super-blocks), January 2026. https://arxiv.org/abs/2601.14277 ↩

  3. J. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978), 2023; E. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323), 2022. https://arxiv.org/abs/2306.00978 Β· https://arxiv.org/abs/2210.17323 ↩

  4. OpenAI, gpt-oss-120b (β€œpost-trained with MXFP4 quantization of the MoE weights”, 20b in 16 GB, Apache 2.0), August 2025. https://huggingface.co/openai/gpt-oss-120b; Moonshot AI, Kimi K3 (β€œMXFP4 weights / MXFP8 activations (quantization-aware training)” from SFT onward), July 2026. https://huggingface.co/moonshotai/Kimi-K3; NVIDIA, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (β‰ˆ 21.6 GB of safetensors; β€œthe NVFP4 release is the recommended path” for deployment), August 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 ↩ ↩2

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.