🛠️ Scenario A: Dev Lab (GPU PC or unified memory)
You are a solo developer, a hobbyist (homelab), or a very small business that wants to test autonomous agents without immediately investing €5,000 to €10,000 in a dedicated AI machine.
This first blueprint covers two solo AI lab realities:
- Option A1 — Standard PC with 24 GB GPU: excellent for 8B–14B models, but penalized by CPU Offloading as soon as a model exceeds VRAM.
- Option A2 — Laptop or station with 64–128 GB unified memory: often the best comfort for a solo AI developer in 2026, because large quantized models can fit in a single memory pool without constant PCIe round-trips.
🏗️ Hardware architecture
Option A1 — Standard PC with 24 GB GPU
- Machine: A standard tower PC.
- Processor (CPU): A modern processor (AMD Ryzen 9 or Intel Core i9).
- System memory (RAM): 64 GB DDR5 RAM (essential for offloading — DDR4 would completely throttle performance; buy early: DRAM prices have risen every quarter since early 2026 and a shortage is expected for 2027)1.
- Graphics card (GPU): A single consumer NVIDIA card with 24 GB VRAM or more (e.g. a used RTX 3090 or 4090; the 32 GB RTX 5090, selling for $5,000 and up since September 2026 (see 🧩 Stations Multi-GPU), falls outside a lab’s budget — new 24 GB+ GeForce supply is scarce in Q4 2026).
Estimated budget (Q4 2026): between €1,500 and €2,500 with a used RTX 3090/4090; above €6,000 with a new RTX 5090 at observed prices. The 64 GB of DDR5 and the used 24 GB GPU are the two moving line items: buy them early (dated prices, see 💰 Comparaison TCO).
Option A2 — Laptop / station with 64–128 GB unified memory
- Machine: MacBook Pro Max, entry-level Mac Studio, or mini-station APU with large unified memory. Named options in Q4 2026: Mac mini M5 Pro 64 GB (307 GB/s, Thunderbolt 5) at the entry level, Mac Studio M5 Max 64 GB (460 to 614 GB/s depending on GPU, €3,659 incl. VAT), Framework Desktop 64 GB (€2,209 incl. VAT, often out of stock), or DGX Spark 64 GB ($4,999 from OEMs, native CUDA; see APU & Mémoire Unifiée)23.
- Memory: 64 to 128 GB unified memory.
- Engine: MLX (Ollama ≥ 0.40’s default engine on Apple Silicon; experimental Splash alternative for Qwen3.8) / llama.cpp (default port 9931 since October 2026) / Ollama depending on platform4.
- Ideal case: solo developer who wants to test 30B–70B quantized models with better interactive comfort than DDR5 CPU offloading.
Estimated budget (Q4 2026): between €2,200 (Framework Desktop 64 GB, often out of stock) and €3,700 (Mac Studio M5 Max 64 GB, €3,659 incl. VAT) for 64 GB; from €3,900 to €7,000 for 128 GB (Framework 128 GB, Mac Studio M5 Max 128 GB, DGX Spark 128 GB)23. More expensive than a used gaming PC, but much more coherent if your goal is to regularly work with large local models. Soldered memory follows the DRAM increase: October 2026 prices, to be re-checked at purchase time (see 💰 Comparaison TCO).
⚙️ Software stack
- Inference engine: Ollama or llama.cpp compiled with CUDA support.
- Model format: GGUF in Q4_K_M quantization.
On this machine, an 8B to 30B class model (e.g. Qwen3.8-27B ≈ 18 GB in Q4, Granite 4.2 8B, or a light MoE such as Nemotron 3.5 Lightning 30B-A3B ≈ 22 GB in NVFP4) will fit entirely in the graphics card’s 24 GB VRAM56. Do not rely on the name of Nemotron 3.5 Bolt 74B-A9B 24GB (October 2026): despite its 2-bit experts, NVIDIA only publishes a recipe starting from a 32 GB GPU or a 64 GB RTX/DGX Spark (28 GB recipe, 64K context). Neither Ollama nor the NVIDIA model card (Windows and Linux) supports Apple Silicon, which rules out option A2 on Mac7. Qwen3.8-27B (Apache 2.0, 262k context, vision and tools) has been the reference 24 GB model since August 2026; Meta publishes Muse Glimmer 30B (Apache 2.0) with explicit VRAM tiers — 64 GB at full precision, 32 GB and 24 GB in K-Quant8. You will get high performance — typically 50 to 100 tokens/s for an 8B–14B, 30 to 60 t/s for a 27B Q4 — more with speculative decoding (Meta announces ≈ 230 t/s on RTX 5090 for Muse Glimmer 30B with DFlash, vendor figure)8.
But what happens if you want to test a heavy 70B dense model (Llama 3.3 70B, still very widespread) or a 120B MoE such as gpt-oss-120b9?
🧠 The mechanism: CPU Offloading
A 70B model quantized in Q4 weighs about 40 GB. It is physically impossible to fit it into a 24 GB card. This is where CPU Offloading comes in.
Rather than giving up with an Out Of Memory (OOM) error, the llama.cpp engine splits the model:
- It loads as many neural network layers as possible into the GPU’s ultra-fast VRAM (about 20 to 22 GB to leave headroom for context).
- It places the remaining layers (about 18 to 20 GB) in the motherboard’s system RAM.
⚠️ The performance wall
During response generation (Decoding), the layers left in RAM are computed by the processor, at the speed of DDR5 RAM; only the activations travel over the PCIe bus.
As explained in the chapter on VRAM vs RAM, classic RAM is physically capped at about 80–100 GB/s. The result is immediate: generation speed collapses.
On an RTX 4090 paired with 64 GB DDR5, a 70B Q4 model in CPU Offloading will generate at most 4 to 5 tokens per second depending on the share of offloaded layers and the RAM: theoretical bound of ~4–5 t/s if half the weights travel through DDR5 (~100 GB/s, see memory bandwidth — the ~2.5 t/s bound assumes the whole model goes through RAM), the measured throughput remaining lower; the mechanism (-ngl option, GPU vs CPU layers) is documented by llama.cpp and Ollama1011. It is readable (slightly below human reading speed), but unsuitable for serving a reactive application or several concurrent users.
Why unified memory changes the experience
On a unified memory machine, model weights are not split between fast VRAM and slow RAM connected by PCIe. CPU, GPU, and accelerators share the same memory pool. Bandwidth remains lower than a high-end NVIDIA card (RTX 5090: 1.8 TB/s; Mac Studio M5 Max: 614 GB/s, M5 Ultra: 1.2 TB/s)12, but it avoids the worst trap of the standard PC: computing part of the model at the slow speed of DDR5 RAM.
For a solo developer, this often makes the difference between “I can test a quantized 70B and reason comfortably” and “I watch tokens arrive one by one”.
📋 The architect’s verdict
✅ When to use this blueprint?
- To learn and prototype applications (RAG, agents) on small models (8B/14B) that fit 100% in VRAM.
- To run background tasks (batch processing, overnight summarization of long documents) with a 70B model, where the user is not waiting for the response live on screen.
- For a solo developer with a 64–128 GB unified memory machine who wants to test larger models without building a server appliance.
❌ When to avoid this blueprint?
- If you need to deploy an internal API for more than 2 concurrent collaborators. CPU Offloading handles concurrency very poorly: beyond one request at a time, response time collapses.
- If employee usability comfort is an absolute priority.
For daily SME use with 70B models without suffering this heavy transfer penalty, you need to change the hardware paradigm and move from a solo workstation to a service machine. That is the subject of the next blueprint: The Unified Appliance (APU/Mac Unified Memory).
📚 Sources and references
Footnotes
-
TrendForce, press release of 30 September 2026 (contract DRAM prices up 10–15% in Q4 2026). https://www.trendforce.com/presscenter/news/20260930-13258.html ↩
-
Apple, Mac Studio — Apple Store France (Mac Studio M5 Max 64 GB €3,659 incl. VAT), captured 2026-10-09. https://www.apple.com/fr/shop/buy-mac/mac-studio · Apple Newsroom, Apple unveils a more powerful Mac mini featuring the all-new M6 and M5 Pro (Mac mini M5 Pro: up to 64 GB, 307 GB/s, Thunderbolt 5), August 2026. https://www.apple.com/newsroom/2026/08/apple-unveils-a-more-powerful-mac-mini-featuring-the-all-new-m6-and-m5-pro/ ↩ ↩2
-
Framework, Framework Desktop — AMD Ryzen AI Max+ 395 (64 GB €2,209 incl. VAT, 128 GB €3,889 incl. VAT, out of stock), captured 2026-10-09. https://frame.work/fr/fr/products/desktop-diy-amd-aimax300 ↩ ↩2
-
Ollama, Release v0.40.0 (MLX used by default on Apple Silicon), October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.0 · ggml-org, llama.cpp — Release b11521 (
llama-serverdefault port changed from 8080 to 9931), 9 October 2026. https://github.com/ggml-org/llama.cpp/releases/tag/b11521 · incoai, Splash (experimental Metal engine, M3+ and ≥ 36 GB, Qwen3.8; throughput announced by the publisher, not independently verified), 18 September 2026. https://github.com/incoai/splash ↩ -
Qwen, Qwen3.8-27B (dense, Apache 2.0, 262k context, vision and tools; BF16 ≈ 56 GB, Q4 via Ollama ≈ 18 GB), August 2026. https://huggingface.co/Qwen/Qwen3.8-27B · Ollama, qwen3.8. https://ollama.com/library/qwen3.8 ↩
-
NVIDIA, Nemotron-3.5-Lightning-30B-A3B (BF16 / NVFP4 ≈ 22 GB / GGUF, OpenMDW 1.1 license), August 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↩
-
NVIDIA, NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF (GGUF 22 GB + mmproj 1.73 GB; llama.cpp “28GB” recipe for a 64 GB RTX/DGX Spark or 32 GB+ GPU, context 65,536 tokens, q8_0 KV cache; “32GB” recipe for a 128 GB Spark or 48 GB+ GPU; supported systems: Windows, Linux), published 2026-10-15, accessed 2026-11-20; Ollama, nemotron-3.5-bolt (“Apple Silicon / Metal is not currently supported” for 74B-A9B), accessed 2026-11-20. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF · https://ollama.com/library/nemotron-3.5-bolt ↩
-
Meta, Muse Glimmer 30B (Apache 2.0, 29.6B including vision encoder; tiers 64 GB full precision, 32 GB K-Quant-Dynamic, 24 GB K-Quant-17GB; 74.9 tok/s without speculation and 233.4 tok/s with DFlash on RTX 5090, vendor figures), August 2026, re-read on 2026-10-10. https://huggingface.co/meta-models/Muse-Glimmer-30B ↩ ↩2
-
OpenAI, gpt-oss-120b (117B MoE, 5.1B active, MXFP4, Apache 2.0; ≈ 60–70 GB), August 2025, re-read on 2026-10-09. https://huggingface.co/openai/gpt-oss-120b ↩
-
ggml-org, llama.cpp — llama-server README (
-ngl/--n-gpu-layersoption: number of layers placed in VRAM, the rest computed on CPU), re-read on 2026-10-10. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md ↩ -
Ollama Documentation, FAQ — GPU layer offloading and partial CPU inference (Performance penalty during RAM offloading), 2026. https://docs.ollama.com/faq ↩
-
Apple Newsroom, Apple introduces new Mac Studio with M5 Max and M5 Ultra (M5 Max 614 GB/s, M5 Ultra 1.2 TB/s), 25 August 2026. https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ · NVIDIA, GeForce RTX 5090 (32 GB GDDR7, 512-bit bus). https://www.nvidia.com/fr-fr/geforce/graphics-cards/50-series/rtx-5090/ ↩
Incoming links
- 🏗️ Possible Architectures
- 🏢 Scenario B: SME Appliance (Unified Memory)
- 💰 TCO Comparison: On-Premise vs Cloud API
- 🔒 Sovereignty & Privacy
- 🖥️ Index — Hardware
- 🗺️ Choosing your local model
- 🤖 On-Premise Agents & Assistants
- 🚀 Zero to Hero Index
- Aider
- Cursor CLI
- Jan.ai
- Khoj
- Offloading
- Ollama
- On-Premise (AI)
- Open WebUI
- OpenHuman
- Q4 Quantization
- RAM
- Zero to Hero: On-Premise AI