🗺️ Choosing your local model
Three questions before choosing
Before looking at a leaderboard, answer these three questions in order:
1. What is your primary task?
| Task | Recommended model profile |
|---|---|
| Chat / general assistant | Generalist instruction-tuned model |
| Document RAG (in French) | Strong instruction-following model, good long context |
| Custodian agent / code editing | 27–30B agentic generalist (Qwen3.8-27B, Muse Glimmer 30B, Granite 4.2) — see “Specializations” below |
| Summarization, extraction, classification | Compact fast model, 7–8B often sufficient |
| Reasoning / complex calculation | “Thinking” model (built-in chain-of-thought) |
2. How much VRAM do you have?
See Quantization to calculate exact footprint. In Q4_K_M, approximate rule1:
| Available VRAM | Accessible model size |
|---|---|
| 8–12 GB | 7–8B |
| 16–24 GB | 14B (16 GB) — up to ~30B in Q4 (24 GB, e.g. Qwen3.8-27B ≈ 18 GB); despite its name, Nemotron 3.5 Bolt 74B-A9B “24GB” has no NVIDIA recipe for a 24 GB card (the lightest: 28 GB, for a GPU with 32 GB or more)2 |
| 48 GB | 32–34B comfortably |
| 80 GB (H100) | 70B in FP8/Q8 (~70 GB, short context) or up to ~120B in Q4 (≈ 0.55–0.6 byte/parameter, short context)3 |
| 128–160 GB (APU) | 70B Q8 or ~150–230B in Q4 depending on the memory allocatable to the GPU and the KV headroom |
3. How many concurrent users?
The more users, the smaller the model must be to leave VRAM for concurrent KV Cache. A 70B model that responds perfectly to one user can collapse at five.
The open-weights landscape (Q4 2026)
The market has stabilized around a few dominant families. Here is how to read them.
Llama 3.x (Meta)
The generalist reference of previous generations. Llama 3.1/3.3 models are available in 8B, 70B, and 405B. Well documented, supported by all engines (Ollama, vLLM, TensorRT-LLM), with a permissive commercial license.
- Llama 3.3 70B: long the best quality/size ratio for SME use cases; in Q4 2026, it remains a safe and very well-supported choice, but it is outperformed at equal size by 27–30B generalists (Qwen3.8-27B, Muse Glimmer 30B). Strong instruction following, reasoning, and multilingual (including French).
- Llama 3.1 8B: good for constrained workstations or simple tasks. Visible limits on complex reasoning tasks.
- Llama 3.1 405B: requires a multi-GPU cluster (Scenario D). Performance close to frontier models on general tasks.
Llama 4 (Meta) — Scenario D only
Natively multimodal (text + image), MoE architecture, released in April 20254. There is still no Llama 4 usable on a desktop machine, and Meta has not released any Llama since: its open line is now called Muse Glimmer 30B (Apache 2.0, text + image, ~24 GB in 4-bit)5.
- Llama 4 Scout (109B total / 17B active, 16 experts): 10M token context. Fits on a single H100 with int4 quantization. Relevant only for Scenario D (datacenter).
- Llama 4 Maverick (400B total / 17B active, 128 experts): 1M token context. Requires a full DGX host in FP8 or BF16. Performance comparable to frontier models on STEM benchmarks.
Qwen3.8 / Qwen3 (Alibaba)
The most versatile family in the open-weights landscape, with excellent multilingual coverage (including French). As of Q4 2026:
- Qwen3.8-27B (dense, Apache 2.0, vision + video, 262k tokens): ~18 GB in Q4 via Ollama, the default choice for a 24 GB GPU and for code agents (SWE-bench Pro 61.7)3.
- Qwen3-30B-A3B (MoE, Apache 2.0): 3B active, ~18 GB in Q4, excellent throughput on APUs; same class as Nemotron 3.5 Lightning 30B-A3B (NVIDIA, official NVFP4 ≈ 22 GB, commercial OpenMDW 1.1 license)6.
- Qwen3.8-Flash-Next and Qwen3.8-2.4T-A95B (“Qwen3.8-Max”): flagships under custom licenses (qwen-community-1.0, qwen3.8-max), to be read before any commercial use; datacenter size.
DeepSeek (DeepSeek AI)
- DeepSeek V4 / V4.1 (MIT): the current generation. V4-Flash (284B with 13B active for the preview, 304B for the 0731 revision, 1M tokens) and V4.1-Flash (552B + 196B of conditional memory, 8 to 16B active, FP4 KV cache) replace V3 and R1; reasoning is built in (adjustable effort), no more separate “R” model. V4.1-Flash and V4-Pro are datacenter size (8× 80 GB GPU node minimum); V4-Flash (~300 GB in FP8, ~150–170 GB in 4-bit) also fits on a 256–384 GB desktop cluster (Scenario C)7.
- DeepSeek-R1 / V3 (2025): still available and well supported, but outclassed at equal cost.
Mistral / Mixtral (Mistral AI)
- Mistral Small 4 (119B) and Mistral Medium 3.5 (128B): the current open line, with official NVFP4 checkpoints for Small 49. Mistral 7B and Mixtral 8x7B remain usable but date from 2023-2024.
- Mistral Large 4 (~1T MoE, 52B active, multimodal, 1M context): presented on 6 October 2026 in API preview, weights published on 29 October 2026 on Hugging Face (
mistralai/Mistral-Large-4-1T-A52B) under the Mistral Ecosystem Open-Weight license, a modified MIT. Use, modification, fine-tuning and commercial use are free, but a company that resells it as Model-as-a-Service must obtain a separate commercial license from Mistral. Datacenter size10.
Kolibri (Aleph Alpha)
- Kolibri 1 (Aleph Alpha, Germany, 3 October 2026): a 78B-parameter reasoning MoE with 3.46B active per token, under Apache 2.0, with tool calling and a native context of 262k tokens (validated up to 1M). The weights are published in FP8 (~78 GB): the vendor states a minimum of one H200, one B200 or two 80 GB GPUs (A100 / H100), hence a Scenario D-type server; with so few active parameters, throughput remains that of a small model11.
- Two limits to weigh before choosing it: it targets German and English only (French is not a target language), and it is served through a vendor vLLM plugin (
aleph-alpha-inference), with no Ollama build or GGUF announced. Its main appeal is provenance: a European vendor, signatory of the EU GPAI code of practice; the published scores are the vendor’s own, to be confirmed on your corpus (see Evaluate a local model)11.
Nemotron 3.5 (NVIDIA)
- Nemotron 3.5 Bolt 74B-A9B 24GB (15 October 2026, OpenMDW 1.1 license, commercial use allowed): a hybrid Mamba-2/Transformer MoE with 74B parameters, 9B of them active, with vision and 256K context. NVIDIA compressed it from Nemotron 3.5 Super VL and ships it with 2-bit experts (IQ2_XXS) trained by distillation (QAD). The “24GB” refers to the memory footprint targeted on RTX/DGX Spark, not a 24 GB card: the GGUF weighs 22 GB plus 1.7 GB of vision encoder. NVIDIA’s llama.cpp recipes call for 28 GB (64K context, q8_0 KV cache, a GPU with 32 GB or more, or a 64 GB RTX/DGX Spark) or 32 GB (256K context, a GPU with 48 GB or more, or a 128 GB Spark)2.
- Appeal for a single workstation: on RTX Spark, with its drafter, NVIDIA measures it at 84 tokens/s, versus 28 for Qwen3.8-27B in Q4 but 97 for Qwen3.6-35B-A3B and 101 for Gemma 4 26B-A4B. NVIDIA rates it ahead of these two MoEs on many agent benchmarks (vendor figures). Limits as of Q4 2026: no Apple Silicon support in Ollama, and vLLM only through a nightly image. Validate it on your golden dataset before preferring it to a 27–30B in Q42.
Phi-4 / Phi-3 (Microsoft)
Compact models (3.8B–14B) with high reasoning quality for their size. Interesting for desktop use with little VRAM.
- Phi-4 14B: performance close to some 70B models on reasoning and code tasks, for 8 GB VRAM in Q4.
Model → on-premise scenario mapping
| Scenario | Typical hardware | Recommended model | Use case |
|---|---|---|---|
| A — Dev Lab | PC with 24 GB GPU (+ offloading beyond) or 64 GB unified memory | Qwen3.8-27B Q4 (~18 GB)3, Granite 4.2 8B12 or Gemma 4 E4B13 | Solo dev, testing, prototyping |
| B — SME Appliance | APU 128 GB unified memory | Qwen3.8-27B3 or Muse Glimmer 30B5 (Q8 possible); Nemotron 3.5 Lightning 30B-A3B NVFP4 (~22 GB, 256k) for throughput6 | Team assistant, document RAG |
| C — Desktop Cluster | 2–4 Thunderbolt machines | GLM-5.3-Flash (320B / 18B active, MIT)14 or DeepSeek V4-Flash (MIT)7 | Advanced SME, highly capable model |
| D — Datacenter | Multi-H100 / MI300X | DeepSeek V4.1-Flash / V4-Pro (MIT)7, MiMo-V2.6-Pro (1.02T / 42B active) and MiMo-V2.6-Flash (309B / 15B active), MIT, text + image + video + audio, since September 202615, GLM-5.3 (custom license)14, Kimi K3 (native MXFP4, custom license)16, Qwen3.8-2.4T-A95B (custom license); Llama 4 Scout / Maverick remain valid4 | Production 50+ users, SLA, multimodal |
Specializations: when to choose a coder model?
Generalist models (Llama, general Qwen) can write code, but they are not built to reliably modify an existing repository. A custodian agent that must perform precise search-and-replace in Markdown or code needs a model trained for agentic code editing.
Since 2026, 27–30B generalist models trained for agentic use (Qwen3.8-27B, Muse Glimmer 30B, Granite 4.2) have replaced the dedicated “Coder” variants: they outperform the 2024 Coder models on SWE-bench, and the coder / generalist distinction has blurred.
Practical rule (Q4 2026):
| Use | Minimum model | Recommended model |
|---|---|---|
| Code completion in an IDE | Granite 4.2 8B12 | Qwen3.8-27B3 |
| Custodian agent (controlled corrections) | Qwen3.8-27B3 | Muse Glimmer 30B5 |
| Autonomous agent (regular maintenance) | Qwen3.8-27B (Apache 2.0, 262k)3 | Muse Glimmer 30B (Apache 2.0, SWE-bench Verified 76.0)5 or Granite 4.2 30B (Apache 2.0)12 |
Reasoning models: when do you need them?
“Thinking” models (Qwen3.83, Granite 4.212, DeepSeek V4.17 — all with a switchable reasoning mode or an adjustable effort level) generate internal reasoning before the response. They are useful for:
- mathematical or logical problems;
- multi-step analyses (due diligence, audit);
- tasks where a reasoning error is costly.
In exchange:
- TTFT is longer (the model “thinks” before answering);
- reasoning tokens consume context and VRAM;
- they are oversized for simple tasks (extraction, classification, chat).
How to read a leaderboard without getting it wrong
Public rankings (Arena, Artificial Analysis, SWE-bench Pro, HELM) are useful for initial orientation, but do not replace your own tests.
What leaderboards still tell you usefully:
- Hugging Face: the Open LLM Leaderboard has been archived since 2025; use the Hub to check license, file sizes, and available quantizations, and Artificial Analysis (checking the index version) for a composite view18.
- SWE-bench Pro V2 (Scale)17: the most representative for code agents (real GitHub issues, protocol frozen since September 2026).
Where do open weights stand against closed models (October 2026)? On Artificial Analysis’ Intelligence Index v4.3, Claude Opus 5.5 scores 58, Claude Fable 5.1 and GPT-6 Astra 53; the best open-weights models follow at 46 (MiMo-V2.6-Pro, MIT), 45 (GLM-5.3) and 44 (Kimi K3)19. On arena.ai, no open model appears in the top 15 as of 2026-10-08; the first, kimi-k3-max, ranks 16th20. The gap is real, but it closes within a few quarters with each new open generation. For an on-premise reader, the question is therefore not the podium but the level sufficient for the task, measured on your own data: a Qwen3.8-27B that passes your golden dataset is worth more than a frontier model you can neither host nor audit.
Selection checklist
Before downloading a model:
- Does the license allow your use? Apache 2.0 / MIT / OpenMDW 1.1: yes. Llama Community, Kimi K3 License, glm-5.3, qwen3.8-max, qwen-community-1.0: read the thresholds (MAU, MaaS revenue) before deploying1416.
- Are the weights actually downloadable? An Ollama
:cloudtag (Kimi K3, GLM-5.3) means hosted inference, not on-premise21. - Does the model fit in your VRAM with the target quantization + KV Cache headroom?
- Does the target inference engine support it? (GGUF for Ollama, safetensors for vLLM)
- Do community evaluations exist for your language? (French is less covered than English)
- Do you have a golden dataset to test on your real data?
- For an agent: do you have a 27–30B agentic model (Qwen3.8-27B, Muse Glimmer 30B, Granite 4.2 30B), not a 7B generalist?
See also
- 🧪 Evaluating a local model — test protocol, KPIs, golden dataset
- 🗜️ Quantization — calculate VRAM footprint
- ⚙️ Inference engines — choose the right engine by use case
- MoE — understand Mixture of Experts architectures
- LLM benchmark
📚 Sources and references
Footnotes
-
J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (Q4_K_M size reduction ≈ 69%, i.e. ≈ 0.5 byte/parameter + KV Cache headroom), arXiv:2601.14277, January 2026. https://arxiv.org/abs/2601.14277 ↩
-
NVIDIA, NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF (74B / 9B active, IQ2_XXS experts via QAD, 256K context, OpenMDW 1.1; GGUF 22 GB + mmproj 1.73 GB; llama.cpp recipes at 28 GB for a 64 GB RTX/DGX Spark or a 32 GB+ GPU and at 32 GB for a 128 GB Spark or a 48 GB+ GPU), published 2026-10-15, accessed 2026-11-20; NVIDIA, NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB (safetensors checkpoint, vLLM via a
vllm/vllm-openai:nightlyimage); NVIDIA Research, NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (AIPerf SPEED-Bench-Coding on a 128 GB RTX Spark: 83.99 tok/s versus 96.63 for Qwen3.6 35B-A3B, 100.56 for Gemma 4 26B-A4B and 28.26 for Qwen3.8 27B, all in UD-Q4_K_M), 2026-10-15; Ollama, nemotron-3.5-bolt (25 GB, Apple Silicon / Metal not supported for 74B-A9B), accessed 2026-11-20. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF · https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/ · https://ollama.com/library/nemotron-3.5-bolt ↩ ↩2 ↩3 -
Qwen, Qwen3.8-27B (dense, Apache 2.0, 262k context; BF16 ≈ 56 GB, Q4 via Ollama ≈ 18 GB — i.e. ~2 bytes/parameter in BF16 and ~0.6 in Q4), August 2026. https://huggingface.co/Qwen/Qwen3.8-27B ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8
-
Meta AI, The Llama 4 herd: natively multimodal AI innovation (Scout 109B/17B active, Maverick 400B/17B active, MoE architecture), April 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ ↩ ↩2
-
Meta, Muse Glimmer 30B (Apache 2.0, text + image, SWE-bench Verified 76.0), August 2026. https://huggingface.co/meta-models/Muse-Glimmer-30B ↩ ↩2 ↩3 ↩4
-
NVIDIA, Nemotron-3.5-Lightning-30B-A3B (BF16 / NVFP4 ≈ 22 GB / GGUF, OpenMDW 1.1 license, 256k context on an 80 GB GPU), August 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↩ ↩2
-
DeepSeek AI, DeepSeek-V4.1-Flash (MIT, 552B + 196B of Engram conditional memory, 8–16B active, FP4 KV cache 890 bytes/token), September 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash; DeepSeek AI, DeepSeek-V4-Flash-0731 (MIT, ~304B, 1M tokens), 2026-07-31. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 ↩ ↩2 ↩3 ↩4
-
DeepSeek AI, DeepSeek-V3 (671B, 37B active, native FP8 weights). https://huggingface.co/deepseek-ai/DeepSeek-V3; unsloth, DeepSeek-V3-GGUF (Q4_K_M: 9 files, ≈ 404 GB — all experts resident in VRAM), January 2025. https://huggingface.co/unsloth/DeepSeek-V3-GGUF ↩
-
Mistral AI, Hugging Face repositories (Mistral-Small-4-119B-2603 and its NVFP4 variant, Mistral-Medium-3.5-128B), captured 2026-10-09. https://huggingface.co/mistralai ↩
-
Mistral AI, Mistral Large 4 (API preview), 2026-10-06. https://mistral.ai/news/mistral-large-4 · Mistral AI, announcement of the weights release (Hugging Face
mistralai/Mistral-Large-4-1T-A52B, Mistral Ecosystem Open-Weight license: modified MIT, separate commercial license for Model-as-a-Service offerings), 2026-10-29. https://mistral.ai/news/mistral-large-4-oss/ · https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/LICENSE ↩ -
Aleph Alpha, Kolibri-1 (78B MoE, 3.46B active, Apache 2.0, German and English, native context 262,144 tokens validated up to 1,048,576, FP8 weights ≈ 78 GB; minimum 2× A100 80 GB, 2× H100 SXM5, 1× H200, 1× B200 or 1× B300; vLLM plugin
aleph-alpha-inference), 2026-10-03. https://huggingface.co/Aleph-Alpha/Kolibri-1 ; Aleph Alpha, Kolibri has landed: a sovereign open-weight model, 2026-10-03. https://aleph-alpha.com/en/blog/kolibri-has-landed-a-sovereign-open-weight-model/ ↩ ↩2 -
IBM, Granite 4.2 30B (Apache 2.0, OpenAI-format tool calling), 2026-08-25. https://huggingface.co/ibm-granite/granite-4.2-30b ↩ ↩2 ↩3 ↩4
-
Google, Gemma 4 31B IT (Apache 2.0; E2B / E4B / 26B-A4B / 31B family), April 2026. https://huggingface.co/google/gemma-4-31b-it ↩
-
Z.ai, GLM-5.3-Flash (320B MoE, 18B active, MIT), 2026-08-26. https://huggingface.co/zai-org/GLM-5.3-Flash; Z.ai, GLM-5.3 (
glm-5.3license). https://huggingface.co/zai-org/GLM-5.3 ↩ ↩2 ↩3 -
Xiaomi, MiMo-V2.6-Pro-MOPD and MiMo-V2.6-Flash-MOPD (1.02T / 42B-active and 309B / 15B-active MoEs, 1M context, omni, MIT license; the publisher’s vLLM recipes at TP 8 for Pro and TP 4 for Flash, SGLang recipe for Pro on two nodes; MOPD update of 2026-09-27 against repeated tool calls), Hugging Face. https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD · https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD ↩
-
Moonshot AI, Kimi K3 (native MXFP4; Kimi K3 License: separate agreement above $20M of MaaS revenue over 12 months). https://huggingface.co/moonshotai/Kimi-K3 ↩ ↩2
-
Scale AI, SWE-Bench Pro V2 (642 tasks, 11 repositories, 2026-09-22). https://labs.scale.com/leaderboard/swe_bench_pro_public_v2; Vals AI, SWE-bench Verified (runs stopped on 2026-09-01, top > 96%). https://www.vals.ai/benchmarks/swebench ↩ ↩2
-
Hugging Face, Open LLM Leaderboard — archive. https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/archive; Artificial Analysis, Intelligence Index v4.3 (2026-09-07). https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 ↩
-
Artificial Analysis, Intelligence Index — model ranking (Claude Opus 5.5 58, Claude Fable 5.1 53, GPT-6 Astra 53, MiMo-V2.6-Pro 46, GLM-5.3 45, Kimi K3 44; index v4.3 or revision 4.3.x), captured 2026-10-10. https://artificialanalysis.ai/leaderboards/models; Artificial Analysis, Claude Opus 5.5 (58, “the highest score we have measured by several points”), 2026-09-22. https://artificialanalysis.ai/articles/claude-opus-5-5 ↩
-
Arena, Text Leaderboard (multi-turn human preference ranking; snapshot of 2026-10-08: no open-weights model in the top 15, kimi-k3-max at rank 16). https://arena.ai/leaderboard/text ↩
-
Ollama, Library — kimi-k3 (only tag
kimi-k3:cloud, 2.81T parameters: hosted inference) and glm-5.3 (:cloudtag), accessed 2026-10-10. https://ollama.com/library/kimi-k3 · https://ollama.com/library/glm-5.3 ↩