GGUF
π Short definition
Single portable file format bundling weights, metadata, and quantization schema for local inference via llama.cpp/Ollama.
π Detailed definition
GGUF (successor to GGML) packs everything the engine needs: quantized weights, tokenizer, hyperparameters, and metadata. K-quant variants (e.g. Q4_K_M, Q5_K_S) offer different size/quality trade-offs via block-wise quantization schemes.
Major advantages: immediate load without compilation, portability across CPU/GPU/Mac, and native partial offloading to RAM when VRAM is insufficient.
π‘ Why it matters for on-prem AI
De facto standard for workstations, Macs, and homelabs. A large share of Hub models have a GGUF version (often community-made: unsloth, bartowski, ggml-org); on Apple Silicon, Ollama runs supported architectures via MLX rather than GGUF, and vendors first publish official FP8/NVFP4 checkpoints1. Essential for Scenario A (dev lab) and Scenario B (Mac Studio).
β οΈ Common pitfalls
- GGUF/llama.cpp can serve multiple users (continuous batching enabled by default in
llama-server, parallel slots via-np), but without vLLM/SGLangβs paged memory management or multi-GPU parallelism: beyond a handful of concurrent users on a heavy model, per-user throughput collapses faster than with a production GPU engine2. - Quantization variants (Q2 through Q8) differ sharply β Q2 and Q3_K_S clearly degrade quality (a measured loss of 4 points on average on downstream tasks for Q3_K_S on Llama 3.1-8B)3.
π Go deeper
- βοΈ Inference engines (GGUF + llama.cpp vs vLLM in production)
- ποΈ 4-bit & 8-bit quantization (precision schemes behind K-quants)
π See also
Footnotes
-
Ollama, Release v0.40.0 (βModels run on MLX on Apple Silicon by defaultβ), October 2026; NVIDIA, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (official BF16 / NVFP4 checkpoints, GGUF provided via ggml-org), August 2026. https://github.com/ollama/ollama/releases/tag/v0.40.0 Β· https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 β©
-
ggml-org, llama.cpp β llama-server README (βContinuous batchingβ, βParallel decoding with multi-user supportβ,
-cbenabled by default,-np N), read on 2026-10-09. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md β© -
J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv:2601.14277: Q3_K_S loses about 4 points on average on downstream tasks), January 2026. https://arxiv.org/abs/2601.14277 β©