Skip to content

🗜️ Quantization (4-bit & 8-bit)

Quantization reduces the numerical precision of network weights (and sometimes activations) 1.

By moving from native FP16/BF16 precision (2 bytes per parameter) to compressed 8-bit (1 byte), 4-bit (~0.5 effective byte), or lower formats, you divide a model’s VRAM footprint by 2 to 4, enabling local execution of models that would otherwise be beyond your hardware 1.


📐 The Mathematical Physics of Quantization

The most common post-training approach is affine block-wise quantization (often called uniform linear quantization with scale and zero-point) 13. It maps a range of real values (x∈[βmin,βmax]x \in [\beta_{min}, \beta_{max}]) to a low-precision discrete set (qq):

q=round(xS)+Zq = \text{round}\left(\frac{x}{S}\right) + Z

Where:

  • SS (Scale): real scaling factor that stretches or compresses the interval 3.
  • ZZ (Zero-point): integer that aligns the real zero to a discrete value 3.
  • round\text{round}: round to nearest integer.
graph LR
    subgraph "High Precision (BF16)"
        A[-3.14] --> B[0.0] --> C[1.59] --> D[5.82]
    end
    subgraph "Low Precision (INT4)"
        E[-8] --> F[0] --> G[4] --> H[15]
    end
    A -.->|Quantization Formula| E
    B -.->|S & Z| F
    C -.->|S & Z| G
    D -.->|S & Z| H

⚠️ The outlier problem

In an LLM, some activations or weights have abnormally high magnitudes 34. If you quantize an entire tensor with a single scale, SS must cover those extremes: most normal values then end up compressed into a few discrete bins, which strongly degrades quality (perplexity, downstream benchmarks) 34.

Recent LLM quantization research aims to work around this: protecting subsets of weights (AWQ), Hessian-based compensation (GPTQ), Hadamard rotations (MR-GPTQ), or fine-grained FP4 formats (NVFP4/MXFP4) 546.


🗺️ The Quantization Format Landscape

Format choice depends on your target infrastructure (CPU, dedicated GPU, unified memory) and inference engine (llama.cpp, vLLM, TensorRT-LLM).

1. GGUF (llama.cpp): the CPU / unified memory standard

The GGUF format (llama.cpp ecosystem) targets local inference, partial offloading to RAM, and Apple Silicon 78.

  • K-Quants (superblocks): weights are split into 256-value super-blocks, with sub-blocks and multiple scales 8. Variants such as Q4_K_M, Q5_K_M, etc. use mixed precision: sensitive tensors (often attention matrices) stay at higher precision than the rest of the network 8.
  • Ideal for: Mac Studio, x86 Ryzen, CPU-first inference, sovereign deployments without a heavy GPU stack 7.

2. AWQ vs GPTQ (GPU standard)

These formats dominate production GPU inference (vLLM, TGI, etc.) 54.

  • GPTQ (Generative Pre-trained Transformer Quantization): one-shot PTQ that minimizes reconstruction error using the Hessian matrix to compensate rounding layer by layer 5.
  • AWQ (Activation-aware Weight Quantization): identifies roughly 1% of the most “salient” weights via activation magnitude and protects them at higher precision; the rest is compressed to 4-bit 4. In practice, AWQ often matches or beats GPTQ on quality benchmarks at equal size 4.
  • Ideal for: maximum GPU throughput on Nvidia/AMD with dedicated kernels.

3. Microscaling FP4: MXFP4 and NVFP4

Standardized by the OCP (Open Compute Project) and hardware-accelerated on recent GPUs (Blackwell; AMD Instinct MI350 on CDNA 4 — the MI300X stops at FP8) 9610:

  • E2M1 format: 4-bit floating-point numbers (1 sign, 2 exponent, 1 mantissa) 9.
  • MXFP4: E8M0 block scale shared across 32 values 9.
  • NVFP4 (NVIDIA): E4M3 FP8 scale over 16 values, plus a per-tensor scale factor 911.
  • MR-GPTQ (ICLR 2026): GPTQ variant with per-block Hadamard rotations, adapted to FP4 constraints. It substantially improves MXFP4 and approaches NVFP4 accuracy, with up to ~4× end-to-end speedups on RTX 5090 and ~2.2× on B200 vs FP16 — but FP4 is not an automatic upgrade over INT4 without a dedicated method 6.

📊 Trade-off: Size, Perplexity, and downstream quality

Perplexity (PPL) on WikiText-2 measures predictive capability (lower is better). It is useful for comparing quantizations on the same model and tokenizer, but is not sufficient: small PPL gaps can hide regressions in reasoning or instruction-following 811.

GGUF benchmarks — Llama 3.1-8B-Instruct (unified data)

Source: llama.cpp study on a single checkpoint, 13 GGUF formats + F16 baseline 8. PPL = WikiText-2; Avg = unweighted mean of GSM8K, HellaSwag, IFEval, MMLU, TruthfulQA.

FormatEffective bpw (≈)Size reduction (%) 8PPL WikiText-2Avg downstreamComment
F16 (reference)16.0—7.3269.47Baseline
Q8_08.046.97.3369.41Nearly identical to F16
Q6_K~6.159.07.3569.23Conservative
Q5_0~5.265.27.4369.92Best Pareto size/quality compromise 8
Q5_K_M~5.564.47.4069.36Robust 5-bit K-quant
Q4_K_S~4.270.87.6269.17Balanced 4-bit default 8
Q4_K_M~4.569.47.5669.15Common community standard
Q3_K_M~3.575.07.9668.07Usable 3-bit if constrained
Q3_K_S~3.477.28.9665.49Maximum compression, visible loss
Q2_K~2.5———See llama.cpp scoreboard: PPL ~9.75 on Llama 3 8B base 12

💡 The quantization paradox (golden rule)

A classic mistake: preferring a small unquantized model over a large quantized model. Under fixed memory constraints, parameter count often dominates bit width 1213.


📋 The Architect’s Advice

For sovereign on-premise local agent deployments:

  1. SMB standard (CPU / Mac / hybrid): Q4_K_S or Q4_K_M in GGUF via llama.cpp — best documented size/quality/speed compromise for Llama 3.1-8B; step up to Q5_0 if quality margin matters more 8.
  2. Shared GPU inference (vLLM / TensorRT-LLM): FP8 on weights (Hopper/Blackwell/recent RTX) or AWQ INT4 depending on the engine; FP8 cuts VRAM ~×2 with low degradation when well calibrated, without being strictly identical to BF16 14. Option names change fast (vLLM 0.31: fp8 → fp8_per_tensor; SGLang ≥ 0.5.16: NVFP4 via FlashInfer only) and FP4 on workstation GPUs (SM120: RTX 50, RTX PRO 6000) remains partially supported in TensorRT-LLM as of Q4 2026: test your engine/GPU pairing before buying 214.
  3. Avoid Q2 and aggressive Q3 in production: Q3_K_S and Q2_K noticeably degrade reasoning (GSM8K) and perplexity; reserve for extreme RAM constraints 812. This rule applies to a model requantized after the fact. A 2-bit model shipped by the vendor and trained for that format is a different case: Nemotron 3.5 Bolt 74B-A9B (October 2026), whose experts are in IQ2_XXS via QAD distillation, and which NVIDIA considers better than smaller 4-bit models at equal memory. Judge it on your own benchmarks 15.
  4. FP4 (NVFP4/MXFP4): natively accelerated on Blackwell (B200, RTX 50, RTX PRO 6000, DGX Spark); since 2026 it is also a delivery format: Kimi K3 is trained in native MXFP4 (QAT), gpt-oss and Nemotron 3.5 Lightning are published in official MXFP4 / NVFP4 (≈ 22 GB for Lightning), Mistral Small 4 has its NVFP4 repository 16. When the vendor provides this checkpoint, prefer it over a community re-quantization; on pre-Blackwell GPUs, the engine falls back to a slower W4A16 path. Always validate on your business benchmarks 611.

📚 Sources and References

Footnotes

  1. NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (quantization, precision/memory trade-off), November 2023. https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ ↩ ↩2 ↩3 ↩4

  2. DeepSeek AI, DeepSeek-V4.1-Flash (“FP4 main KV caching”, E2M1 + one E4M3 scale per 16 channels, 890 bytes per token, MIT), 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash; vLLM Project, Release v0.31.0 (FlashMLA + compressed NVFP4 KV cache by default on SM100 for DeepSeek-V4.1-Flash), 5 October 2026. https://github.com/vllm-project/vllm/releases/tag/v0.31.0 ↩ ↩2

  3. NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization (outliers, activation quantization), November 2023. https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/ ↩ ↩2 ↩3 ↩4 ↩5

  4. J. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978), 2023. https://arxiv.org/abs/2306.00978 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7

  5. E. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323), 2022. https://arxiv.org/abs/2210.17323 ↩ ↩2 ↩3

  6. S. Egiazarian et al., Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization (MR-GPTQ, ICLR 2026), arXiv:2509.23202. https://arxiv.org/abs/2509.23202 ↩ ↩2 ↩3 ↩4

  7. ggml-org, llama.cpp (GGUF engine, K-quant formats), 2026. https://github.com/ggml-org/llama.cpp ↩ ↩2

  8. J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv:2601.14277), January 2026. https://arxiv.org/abs/2601.14277 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10

  9. J. Chhugani et al., Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction (arXiv:2603.08713), 2026. https://arxiv.org/abs/2603.08713 ↩ ↩2 ↩3 ↩4

  10. AMD, AMD Instinct MI350X GPU — product brochure (CDNA 4: hardware support for FP6 and FP4 OCP MX formats, in addition to FP16 / FP8; the MI300X, CDNA 3, stops at FP8), consulted 2026-10-10. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/amd-instinct-mi350x-gpu-brochure.pdf ↩

  11. NVIDIA Technical Blog, Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache (NVFP4 format), December 2025. https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/ ↩ ↩2 ↩3

  12. ggml-org, llama.cpp perplexity scoreboard (WikiText-2, Llama 3 8B, formats K-quant), 2026. https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md ↩ ↩2 ↩3

  13. T. Dettmers, L. Zettlemoyer, The case for 4-bit precision: k-bit Inference Scaling Laws (arXiv:2212.09720; “4-bit precision is almost universally optimal for total model bits and zero-shot accuracy”, more than 35,000 experiments), 2023. https://arxiv.org/abs/2212.09720; Meta, Muse Glimmer 30B (tiers BF16 64 GB / K-Quant-Dynamic 32 GB, −0.2% / K-Quant 24 GB, −1.0% on average over 15 benchmarks), August 2026. https://huggingface.co/meta-models/Muse-Glimmer-30B ↩ ↩2

  14. SGLang Project, Release v0.5.16 (NVFP4 GEMM via FlashInfer only), 2026. https://github.com/sgl-project/sglang/releases/tag/v0.5.16 · NVIDIA, TensorRT-LLM — Releases (1.3.0 rc26/rc27: NVFP4 W4A16 MoE not supported on SM120), captured 2026-10-09. https://github.com/NVIDIA/TensorRT-LLM/releases ↩

  15. NVIDIA Research, NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (FC1/FC2 experts in IQ2_XXS at 2.0625 bits per element via quantization-aware distillation, Mamba layers in NVFP4, embeddings and attention in Q8_0; “better than similarly sized models that are trained from scratch and quantized to 4 bit”), 2026-10-15. https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/ ↩

  16. Moonshot AI, Kimi K3 (“MXFP4 weights / MXFP8 activations (quantization-aware training)”), July 2026. https://huggingface.co/moonshotai/Kimi-K3; NVIDIA, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (≈ 21.6 GB of safetensors, “the NVFP4 release is the recommended path” for deployment), August 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4; OpenAI, gpt-oss-120b (“post-trained with MXFP4 quantization of the MoE weights”, Apache 2.0), August 2025. https://huggingface.co/openai/gpt-oss-120b; Mistral AI, Hugging Face repositories (Mistral-Small-4-119B-2603 and its NVFP4 variant), checked 2026-10-09. https://huggingface.co/mistralai ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.