🚀 Zero to Hero Index

Welcome to the Local AI “Zero to Hero” guide. This vault is designed as a digital garden: don’t read it linearly. Follow your needs, click on concepts you haven’t mastered yet, and build your expertise step by step.

Goal

Learn how to size, deploy, and understand the hardware and software infrastructure required to run massive AI models (Large Language Models) in a 100% local, private, and high-performance way.

---

🔒 Why run AI on-premises?

You may already use ChatGPT or Claude daily. Why bother hosting a model yourself?

1. Data sovereignty Every prompt sent to a cloud service is processed on third-party servers, often outside the European Union. In medical practices, legal departments, public administration, or industrial companies, some data cannot legally leave your premises — GDPR, professional secrecy, classification. On-premise AI is the only technical answer to that constraint. For a 6-criteria audit grid and the GDPR/AI Act framework, see 🔒 Sovereignty & Privacy.

2. Predictable cost at scale Cloud APIs bill per request. At low volume, that’s negligible. With 50 employees querying an assistant all day, the monthly bill can exceed the amortized cost of a dedicated machine within a few months.

3. Full control: model, behavior, availability On-premises, you choose the model, customize it, run without an Internet connection, and availability does not depend on a third party’s pricing or terms of service.

Reading

This vault gives you the keys to understand these trade-offs and choose the architecture suited to your situation — from a solo developer machine to a sovereign datacenter cluster.

---

🗺️ Vault contents

📁 01 - Foundations (The physics of AI)

Understand why a €4,000 computer can still be too slow for AI.

• 🧠 The Journey of a Prompt (How it works)
• 🏎️ Memory Bandwidth (Memory Bandwidth)
• ⚔️ Unified Memory vs RAM vs VRAM
• 💾 KV Cache and Context
• 🗜️ 4-bit & 8-bit Quantization

📁 02 - Hardware

A catalog of architectures.

• 🖥️ Hardware Index — blueprint-guided tour
• 🧠 APU & Unified Memory (AMD Strix Halo & Mac)
• 🧩 Multi-GPU Workstations (Nvidia, PCIe)
• 🏭 GPU rack servers (1U–HGX, RTX vs datacenter)
• 🌐 AI Networking: RoCE, InfiniBand, and Thunderbolt

📁 03 - Software stack (The engines)

How to bring the silicon to life.

• ⚙️ Inference Engines: vLLM, Ollama, and TensorRT-LLM
• 🌐 AI Clustering: Connecting GPUs with Exo and Ray
• 🧩 RAG & Agents: The knowledge architecture
• 🗺️ Choosing your local model — 2026 families, sizes, specializations, blueprint mapping

📁 04 - Architecture blueprints (Scenarios)

Ready-to-propose configurations for your clients.

• 🛠️ Scenario A: Dev Lab (CPU Offloading)
• 🏢 Scenario B: SME Appliance (Unified Memory)
• 🖥️ Scenario C: Desktop Cluster (Exo & Thunderbolt)
• 🏭 Scenario D: Datacenter (RoCE & Multi-GPU)
• 💰 TCO comparison: On-premise vs Cloud API — break-even by scenario

📁 05 - On-Premise Agents & Assistants

The application layer: AI that knows you, and AI that acts for you.

• 🤖 Overview: two tracks, one sovereignty question
• 🔒 Sovereignty & Privacy — 6-criteria audit grid, GDPR/AI Act
• 🏗️ Possible architectures — assistant, custodian, hybrid
• 🧑‍💼 Personal assistants (AI that learns from your data)
• 🤖 Custodian agents (AI that maintains your vault)
• Custodian agent · Human-in-the-loop · Memory Tree

📁 06 - Practical implementation

How to test and decide before deploying.

• 🧪 Overview: practical protocols
• 🚀 Getting started with Ollama — install, first model, API, basic settings
• 🧪 Evaluating a local model — quality, hallucinations, RAG, agents, KPIs, performance
• 🔒 Local inference security — API auth, network isolation, OWASP LLM Top 10, agents
• ⚙️ Configuring multi-GPU vLLM — tensor parallelism, Ray, production
• 📊 Prometheus + Grafana monitoring — GPU metrics, KV Cache, alerts
• 🔄 Migrating from Ollama to vLLM — cutover without downtime, API compatibility

---

🧭 Which blueprint should you choose?

Have a concrete need? This table points you in the right direction in 30 seconds.

Concurrent users	Model size	Hardware budget	→ Blueprint
1 (dev / test)	8–14B	< €3,500	[[04-blueprints/scenario-a-dev-lab
1 (dev / test)	70B with offloading	< €3,500	[[04-blueprints/scenario-a-dev-lab
2–20 (SME, team)	70B	€4,000–8,000	[[04-blueprints/scenario-b-sme-appliance
2–20 (SME, team)	200B+ / MoE	€10,000–15,000	[[04-blueprints/scenario-c-desktop-cluster
50+ (production)	70B–400B	> €300,000	[[04-blueprints/scenario-d-datacenter
SLA constraint < 500 ms	Any	—	[[04-blueprints/scenario-d-datacenter

Reading

Start with the row that matches your user count, then check that the budget fits. When in doubt, read Scenario A first to understand the mechanisms, then move up to your target scenario.

---

🚶 I’m new here: where do I start?

First visit? Follow this path before exploring freely:

1. LLM (what is a large language model?)
2. LLM inference (how you use it day to day)
3. 🧠 The Journey of a Prompt (what really happens when you talk to it)
4. 🏎️ Memory Bandwidth (why your machine can feel slow)
5. 🛠️ Scenario A: Dev Lab (your first concrete blueprint)

---

📖 Dictionary & concepts

Quick definitions for all technical terms. 👉 AI Glossary