🗜️ Quantization (4-bit & 8-bit)
Quantization reduces the numerical precision of network weights (and sometimes activations) 1.
By moving from native FP16/BF16 precision (2 bytes per parameter) to compressed 8-bit (1 byte), 4-bit (~0.5 effective byte), or lower formats, you divide a model’s VRAM footprint by 2 to 4, enabling local execution of models that would otherwise be beyond your hardware 1.
📐 The Mathematical Physics of Quantization
The most common post-training approach is affine block-wise quantization (often called uniform linear quantization with scale and zero-point) 13. It maps a range of real values () to a low-precision discrete set ():
Where:
- (Scale): real scaling factor that stretches or compresses the interval 3.
- (Zero-point): integer that aligns the real zero to a discrete value 3.
- : round to nearest integer.
graph LR
subgraph "High Precision (BF16)"
A[-3.14] --> B[0.0] --> C[1.59] --> D[5.82]
end
subgraph "Low Precision (INT4)"
E[-8] --> F[0] --> G[4] --> H[15]
end
A -.->|Quantization Formula| E
B -.->|S & Z| F
C -.->|S & Z| G
D -.->|S & Z| H
⚠️ The outlier problem
In an LLM, some activations or weights have abnormally high magnitudes 34. If you quantize an entire tensor with a single scale, must cover those extremes: most normal values then end up compressed into a few discrete bins, which strongly degrades quality (perplexity, downstream benchmarks) 34.
Recent LLM quantization research aims to work around this: protecting subsets of weights (AWQ), Hessian-based compensation (GPTQ), Hadamard rotations (MR-GPTQ), or fine-grained FP4 formats (NVFP4/MXFP4) 546.
🗺️ The Quantization Format Landscape
Format choice depends on your target infrastructure (CPU, dedicated GPU, unified memory) and inference engine (llama.cpp, vLLM, TensorRT-LLM).
1. GGUF (llama.cpp): the CPU / unified memory standard
The GGUF format (llama.cpp ecosystem) targets local inference, partial offloading to RAM, and Apple Silicon 78.
- K-Quants (superblocks): weights are split into 256-value super-blocks, with sub-blocks and multiple scales 8. Variants such as
Q4_K_M,Q5_K_M, etc. use mixed precision: sensitive tensors (often attention matrices) stay at higher precision than the rest of the network 8. - Ideal for: Mac Studio, x86 Ryzen, CPU-first inference, sovereign deployments without a heavy GPU stack 7.
2. AWQ vs GPTQ (GPU standard)
These formats dominate production GPU inference (vLLM, TGI, etc.) 54.
- GPTQ (Generative Pre-trained Transformer Quantization): one-shot PTQ that minimizes reconstruction error using the Hessian matrix to compensate rounding layer by layer 5.
- AWQ (Activation-aware Weight Quantization): identifies roughly 1% of the most “salient” weights via activation magnitude and protects them at higher precision; the rest is compressed to 4-bit 4. In practice, AWQ often matches or beats GPTQ on quality benchmarks at equal size 4.
- Ideal for: maximum GPU throughput on Nvidia/AMD with dedicated kernels.
3. Microscaling FP4: MXFP4 and NVFP4
Standardized by the OCP (Open Compute Project) and hardware-accelerated on recent GPUs (Blackwell; AMD Instinct MI350 on CDNA 4 — the MI300X stops at FP8) 9610:
- E2M1 format: 4-bit floating-point numbers (1 sign, 2 exponent, 1 mantissa) 9.
- MXFP4: E8M0 block scale shared across 32 values 9.
- NVFP4 (NVIDIA): E4M3 FP8 scale over 16 values, plus a per-tensor scale factor 911.
- MR-GPTQ (ICLR 2026): GPTQ variant with per-block Hadamard rotations, adapted to FP4 constraints. It substantially improves MXFP4 and approaches NVFP4 accuracy, with up to ~4× end-to-end speedups on RTX 5090 and ~2.2× on B200 vs FP16 — but FP4 is not an automatic upgrade over INT4 without a dedicated method 6.
📊 Trade-off: Size, Perplexity, and downstream quality
Perplexity (PPL) on WikiText-2 measures predictive capability (lower is better). It is useful for comparing quantizations on the same model and tokenizer, but is not sufficient: small PPL gaps can hide regressions in reasoning or instruction-following 811.
GGUF benchmarks — Llama 3.1-8B-Instruct (unified data)
Source: llama.cpp study on a single checkpoint, 13 GGUF formats + F16 baseline 8. PPL = WikiText-2; Avg = unweighted mean of GSM8K, HellaSwag, IFEval, MMLU, TruthfulQA.
| Format | Effective bpw (≈) | Size reduction (%) 8 | PPL WikiText-2 | Avg downstream | Comment |
|---|---|---|---|---|---|
| F16 (reference) | 16.0 | — | 7.32 | 69.47 | Baseline |
| Q8_0 | 8.0 | 46.9 | 7.33 | 69.41 | Nearly identical to F16 |
| Q6_K | ~6.1 | 59.0 | 7.35 | 69.23 | Conservative |
| Q5_0 | ~5.2 | 65.2 | 7.43 | 69.92 | Best Pareto size/quality compromise 8 |
| Q5_K_M | ~5.5 | 64.4 | 7.40 | 69.36 | Robust 5-bit K-quant |
| Q4_K_S | ~4.2 | 70.8 | 7.62 | 69.17 | Balanced 4-bit default 8 |
| Q4_K_M | ~4.5 | 69.4 | 7.56 | 69.15 | Common community standard |
| Q3_K_M | ~3.5 | 75.0 | 7.96 | 68.07 | Usable 3-bit if constrained |
| Q3_K_S | ~3.4 | 77.2 | 8.96 | 65.49 | Maximum compression, visible loss |
| Q2_K | ~2.5 | — | — | — | See llama.cpp scoreboard: PPL ~9.75 on Llama 3 8B base 12 |
💡 The quantization paradox (golden rule)
A classic mistake: preferring a small unquantized model over a large quantized model. Under fixed memory constraints, parameter count often dominates bit width 1213.
📋 The Architect’s Advice
For sovereign on-premise local agent deployments:
- SMB standard (CPU / Mac / hybrid): Q4_K_S or Q4_K_M in GGUF via llama.cpp — best documented size/quality/speed compromise for Llama 3.1-8B; step up to Q5_0 if quality margin matters more 8.
- Shared GPU inference (vLLM / TensorRT-LLM): FP8 on weights (Hopper/Blackwell/recent RTX) or AWQ INT4 depending on the engine; FP8 cuts VRAM ~×2 with low degradation when well calibrated, without being strictly identical to BF16 14. Option names change fast (vLLM 0.31:
fp8→fp8_per_tensor; SGLang ≥ 0.5.16: NVFP4 via FlashInfer only) and FP4 on workstation GPUs (SM120: RTX 50, RTX PRO 6000) remains partially supported in TensorRT-LLM as of Q4 2026: test your engine/GPU pairing before buying 214. - Avoid Q2 and aggressive Q3 in production:
Q3_K_SandQ2_Knoticeably degrade reasoning (GSM8K) and perplexity; reserve for extreme RAM constraints 812. This rule applies to a model requantized after the fact. A 2-bit model shipped by the vendor and trained for that format is a different case: Nemotron 3.5 Bolt 74B-A9B (October 2026), whose experts are in IQ2_XXS via QAD distillation, and which NVIDIA considers better than smaller 4-bit models at equal memory. Judge it on your own benchmarks 15. - FP4 (NVFP4/MXFP4): natively accelerated on Blackwell (B200, RTX 50, RTX PRO 6000, DGX Spark); since 2026 it is also a delivery format: Kimi K3 is trained in native MXFP4 (QAT), gpt-oss and Nemotron 3.5 Lightning are published in official MXFP4 / NVFP4 (≈ 22 GB for Lightning), Mistral Small 4 has its NVFP4 repository 16. When the vendor provides this checkpoint, prefer it over a community re-quantization; on pre-Blackwell GPUs, the engine falls back to a slower W4A16 path. Always validate on your business benchmarks 611.
📚 Sources and References
Footnotes
-
NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (quantization, precision/memory trade-off), November 2023. https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ ↩ ↩2 ↩3 ↩4
-
DeepSeek AI, DeepSeek-V4.1-Flash (“FP4 main KV caching”, E2M1 + one E4M3 scale per 16 channels, 890 bytes per token, MIT), 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash; vLLM Project, Release v0.31.0 (FlashMLA + compressed NVFP4 KV cache by default on SM100 for DeepSeek-V4.1-Flash), 5 October 2026. https://github.com/vllm-project/vllm/releases/tag/v0.31.0 ↩ ↩2
-
NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (outliers, activation quantization), November 2023. https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ ↩ ↩2 ↩3 ↩4 ↩5
-
J. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978), 2023. https://arxiv.org/abs/2306.00978 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7
-
E. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323), 2022. https://arxiv.org/abs/2210.17323 ↩ ↩2 ↩3
-
S. Egiazarian et al., Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization (MR-GPTQ, ICLR 2026), arXiv:2509.23202. https://arxiv.org/abs/2509.23202 ↩ ↩2 ↩3 ↩4
-
ggml-org, llama.cpp (GGUF engine, K-quant formats), 2026. https://github.com/ggml-org/llama.cpp ↩ ↩2
-
J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv:2601.14277), January 2026. https://arxiv.org/abs/2601.14277 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10
-
J. Chhugani et al., Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction (arXiv:2603.08713), 2026. https://arxiv.org/abs/2603.08713 ↩ ↩2 ↩3 ↩4
-
AMD, AMD Instinct MI350X GPU — product brochure (CDNA 4: hardware support for FP6 and FP4 OCP MX formats, in addition to FP16 / FP8; the MI300X, CDNA 3, stops at FP8), consulted 2026-10-10. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/amd-instinct-mi350x-gpu-brochure.pdf ↩
-
NVIDIA Technical Blog, Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache (NVFP4 format), December 2025. https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/ ↩ ↩2 ↩3
-
ggml-org, llama.cpp perplexity scoreboard (WikiText-2, Llama 3 8B, formats K-quant), 2026. https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md ↩ ↩2 ↩3
-
T. Dettmers, L. Zettlemoyer, The case for 4-bit precision: k-bit Inference Scaling Laws (arXiv:2212.09720; “4-bit precision is almost universally optimal for total model bits and zero-shot accuracy”, more than 35,000 experiments), 2023. https://arxiv.org/abs/2212.09720; Meta, Muse Glimmer 30B (tiers BF16 64 GB / K-Quant-Dynamic 32 GB, −0.2% / K-Quant 24 GB, −1.0% on average over 15 benchmarks), August 2026. https://huggingface.co/meta-models/Muse-Glimmer-30B ↩ ↩2
-
SGLang Project, Release v0.5.16 (NVFP4 GEMM via FlashInfer only), 2026. https://github.com/sgl-project/sglang/releases/tag/v0.5.16 · NVIDIA, TensorRT-LLM — Releases (1.3.0 rc26/rc27: NVFP4 W4A16 MoE not supported on SM120), captured 2026-10-09. https://github.com/NVIDIA/TensorRT-LLM/releases ↩
-
NVIDIA Research, NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (FC1/FC2 experts in IQ2_XXS at 2.0625 bits per element via quantization-aware distillation, Mamba layers in NVFP4, embeddings and attention in Q8_0; “better than similarly sized models that are trained from scratch and quantized to 4 bit”), 2026-10-15. https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/ ↩
-
Moonshot AI, Kimi K3 (“MXFP4 weights / MXFP8 activations (quantization-aware training)”), July 2026. https://huggingface.co/moonshotai/Kimi-K3; NVIDIA, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (≈ 21.6 GB of safetensors, “the NVFP4 release is the recommended path” for deployment), August 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4; OpenAI, gpt-oss-120b (“post-trained with MXFP4 quantization of the MoE weights”, Apache 2.0), August 2025. https://huggingface.co/openai/gpt-oss-120b; Mistral AI, Hugging Face repositories (Mistral-Small-4-119B-2603 and its NVFP4 variant), checked 2026-10-09. https://huggingface.co/mistralai ↩
Incoming links
- ⚔️ Unified Memory vs RAM vs VRAM
- 🌐 AI Clustering: Connecting GPUs with Exo and Ray
- 🏭 GPU rack servers
- 🖼️ Multimodality: Hardware Impact (VRAM & KV Cache)
- 🗺️ Choosing your local model
- 🧠 APU & Unified Memory
- 🧩 Multi-GPU Workstations: NVIDIA, PCIe, and VRAM
- 🧪 Evaluate a local model
- 🧪 Practical implementation
- 🚀 Getting started with Ollama
- 🚀 Zero to Hero Index
- Context window
- GGUF
- LLM
- MoE
- Q4 Quantization
- Quantization