Skip to content

GGUF

πŸ“ Short definition

Single portable file format bundling weights, metadata, and quantization schema for local inference via llama.cpp/Ollama.

πŸ“– Detailed definition

GGUF (successor to GGML) packs everything the engine needs: quantized weights, tokenizer, hyperparameters, and metadata. K-quant variants (e.g. Q4_K_M, Q5_K_S) offer different size/quality trade-offs via block-wise quantization schemes.

Major advantages: immediate load without compilation, portability across CPU/GPU/Mac, and native partial offloading to RAM when VRAM is insufficient.

πŸ’‘ Why it matters for on-prem AI

De facto standard for workstations, Macs, and homelabs. A large share of Hub models have a GGUF version (often community-made: unsloth, bartowski, ggml-org); on Apple Silicon, Ollama runs supported architectures via MLX rather than GGUF, and vendors first publish official FP8/NVFP4 checkpoints1. Essential for Scenario A (dev lab) and Scenario B (Mac Studio).

⚠️ Common pitfalls

  • GGUF/llama.cpp can serve multiple users (continuous batching enabled by default in llama-server, parallel slots via -np), but without vLLM/SGLang’s paged memory management or multi-GPU parallelism: beyond a handful of concurrent users on a heavy model, per-user throughput collapses faster than with a production GPU engine2.
  • Quantization variants (Q2 through Q8) differ sharply β€” Q2 and Q3_K_S clearly degrade quality (a measured loss of 4 points on average on downstream tasks for Q3_K_S on Llama 3.1-8B)3.

πŸ“š Go deeper

  1. βš™οΈ Inference engines (GGUF + llama.cpp vs vLLM in production)
  2. πŸ—œοΈ 4-bit & 8-bit quantization (precision schemes behind K-quants)

πŸ”— See also

Footnotes

  1. Ollama, Release v0.40.0 (β€œModels run on MLX on Apple Silicon by default”), October 2026; NVIDIA, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (official BF16 / NVFP4 checkpoints, GGUF provided via ggml-org), August 2026. https://github.com/ollama/ollama/releases/tag/v0.40.0 Β· https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↩

  2. ggml-org, llama.cpp β€” llama-server README (β€œContinuous batching”, β€œParallel decoding with multi-user support”, -cb enabled by default, -np N), read on 2026-10-09. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md ↩

  3. J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv:2601.14277: Q3_K_S loses about 4 points on average on downstream tasks), January 2026. https://arxiv.org/abs/2601.14277 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.