🚀 Getting started with Ollama
Prerequisites
- macOS (Apple Silicon recommended — since Ollama 0.40, October 2026, compatible architectures run on Apple’s MLX engine by default1) or Linux (NVIDIA or AMD GPU, or CPU only)
- Windows: supported via WSL2 or native installer — GPU performance requires CUDA or ROCm drivers
- At least 8 GB RAM (16+ recommended for a comfortable 7B/8B)
- Disk space: 5–50 GB depending on the downloaded model
Installation
macOS / Linux
curl -fsSL https://ollama.com/install.sh | shOllama installs a system service that starts automatically at boot.
Verification:
ollama --version# ollama version is 0.40.x (0.40.16 as of 2026-11-09)
# Is the service running?curl http://localhost:11434/# Ollama is runningWindows
Download the installer from ollama.com/download. The installer configures the background service and adds ollama to PATH.
First model
# Download and run a small recent model (~6.5 GB in Q4_K_M)ollama run qwen3.5:9b
# Or a very light 3B (~2 GB)ollama run llama3.2
# Or a smaller model for a quick test (~2.5 GB)ollama run phi4-mini
# Or a coding model (~18.6 GB)ollama run qwen3-coder:30b# qwen2.5-coder:14b (~9 GB) is still available but is flagged as legacy by Ollama ≥ 0.32The first run downloads the model from ollama.com/library (sizes read from the registry on 2026-10-093). Later runs use the local cache.
To exit the interactive session: /bye or Ctrl+D.
Essential commands
# List downloaded modelsollama list
# See models available online# → https://ollama.com/library
# Download without launchingollama pull qwen3.6:35b # ~22.6 GB
# Remove a model from cacheollama rm llama3.2
# See running processesollama ps
# Service logs (Linux systemd) — there is no `ollama logs` subcommandjournalctl -u ollama --no-pager --follow --pager-end# macOS: cat ~/.ollama/logs/server.logPer-platform log locations are described in Ollama’s troubleshooting page4.
Since Ollama 0.40.2 (October 2026), models downloaded with an earlier version are rewritten in the background on first launch (originals kept as backups); rolling back to a version < 0.40 forces you to re-pull the models — on an appliance, pin the Ollama version5.
OpenAI-compatible API
Ollama exposes a REST API on http://localhost:11434 compatible with the OpenAI format. Any library that uses the OpenAI API works without changes by switching the base URL.
Direct request
curl http://localhost:11434/api/generate \ -d '{ "model": "llama3.2", "prompt": "Explain KV Cache in 3 sentences.", "stream": false }'OpenAI-compatible format (/v1/chat/completions)
curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3.2", "messages": [ {"role": "system", "content": "You are a technical assistant."}, {"role": "user", "content": "What is the difference between prefill and decoding?"} ] }'Python (openai SDK)
from openai import OpenAI
client = OpenAI( base_url="http://localhost:11434/v1", api_key="ollama", # arbitrary value, Ollama does not verify the key)
response = client.chat.completions.create( model="llama3.2", messages=[ {"role": "user", "content": "Summarize unified memory in 2 sentences."} ])print(response.choices[0].message.content)Useful settings
Longer context
By default, the Ollama server uses a 4,096-token window (OLLAMA_CONTEXT_LENGTH); the desktop app picks a value based on VRAM (4k under 24 GiB, 32k between 24 and 48 GiB, 256k above)6. To extend:
# For the whole server (environment variable)OLLAMA_CONTEXT_LENGTH=8192 ollama serve
# Via API (per request)curl http://localhost:11434/api/generate \ -d '{"model": "llama3.2", "prompt": "...", "options": {"num_ctx": 8192}}'
# Via Modelfile (persistent)ollama show llama3.2 --modelfile > Modelfile# Add to the Modelfile:# PARAMETER num_ctx 8192ollama create llama3.2-8k -f ModelfileConcurrent requests
By default, each model handles only one request at a time (OLLAMA_NUM_PARALLEL=1): the second one waits in the queue. Raising this value multiplies the memory reserved for context (parallelism × OLLAMA_CONTEXT_LENGTH); OLLAMA_MAX_LOADED_MODELS (3 × number of GPUs by default) caps the number of models loaded simultaneously6. For several regular users, see ⚙️ Inference engines.
Temperature and generation parameters
curl http://localhost:11434/api/generate \ -d '{ "model": "llama3.2", "prompt": "...", "options": { "temperature": 0.1, "top_p": 0.9, "num_predict": 512 } }'temperature sets how much randomness the model uses (0 = deterministic answers, 1 = more creative) and num_predict the maximum number of tokens generated. JSON does not accept comments: do not paste # into the request body.
Expose Ollama on the local network
By default, Ollama listens only on localhost. To make it reachable from other machines:
# Linux: service environment variableOLLAMA_HOST=0.0.0.0 ollama serve
# Or via systemd (edit /etc/systemd/system/ollama.service)# Environment="OLLAMA_HOST=0.0.0.0"Check performance
# Run a request and measure throughputcurl http://localhost:11434/api/generate \ -d '{"model": "llama3.2", "prompt": "Say hello in 10 languages.", "stream": false}' \ | python3 -c "import sys,json; r=json.load(sys.stdin); \ print(f\"Duration: {r['total_duration']/1e9:.1f}s | \ Tokens generated: {r['eval_count']} | \ Throughput: {r['eval_count']/(r['eval_duration']/1e9):.1f} tok/s\")"Indicative orders of magnitude (community measurements on llama.cpp, mid-2026, not individually sourced; on Apple Silicon, Ollama ≥ 0.40 uses MLX by default and throughputs may differ — measure with the command above and the protocol in 🧪 Evaluate a local model). The Apple rows cover the previous generation, as no measurements have been published for the Mac Studio M5 (theoretical upper bounds in 🏎️ Memory Bandwidth):
| Hardware | 8B Q4 model | 70B Q4 model |
|---|---|---|
| MacBook Pro M4 Pro 48 GB (2024 generation, 273 GB/s) | ~30–50 tok/s | ✗ (~40 GB of weights > ~36 GB allocatable to the GPU by default; upper bound ≤ 6.8 tok/s even after sysctl) |
| Mac Studio M3 Ultra 256 GB (2025 generation, 819 GB/s) | ~50–70 tok/s | ~12–18 tok/s |
| AMD Ryzen AI Max PRO (192 GB, 273 GB/s) | ~25–35 tok/s | ~4–6 tok/s |
| RTX 4090 (24 GB VRAM) | ~50–80 tok/s | Partial offloading |
| CPU only (no GPU) | ~3–8 tok/s | < 2 tok/s |
Next steps
- Choose the right model for your task → 🗺️ Choose your local model
- Move to multi-user production → ⚙️ vLLM in production
- Evaluate quality → 🧪 Evaluate a local model
- Connect an agent or RAG → 🧩 RAG & Agents
Sources and references
Footnotes
-
Ollama, Release v0.40.0 (“Models run on MLX on Apple Silicon by default”), October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.0 ↩
-
Ollama, Release v0.32.0 (
ollamawith no argument launches an agent, default entryglm-5.2:cloud; deprecation warning for legacy agent models), 11 July 2026 · Ollama, Release v0.34.2 (“first-run setup … with options to sign in or continue locally”), 15 September 2026. https://github.com/ollama/ollama/releases/tag/v0.32.0 · https://github.com/ollama/ollama/releases/tag/v0.34.2 ↩ -
Ollama, Library and
registry.ollama.airegistry (manifests:llama3.2= 3B, 2.02 GB;qwen3.5:9b≈ 6.5 GB;qwen3-coder:30b≈ 18.6 GB;qwen3.6:35b≈ 22.6 GB;qwen2.5-coder:14b≈ 9 GB), accessed 2026-10-09. https://ollama.com/library ↩ -
Ollama, Troubleshooting (log locations:
journalctl -u ollama,~/.ollama/logs/server.log), accessed 2026-10-09 ·ollama/ollamarepository,cmd/cmd.go(subcommand list, nologs). https://docs.ollama.com/troubleshooting · https://github.com/ollama/ollama ↩ -
Ollama, Release v0.40.2 (models “upgraded in the background the first time you run them”, backups kept, re-pull required when rolling back to < 0.40), 8 October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.2 ↩
-
Ollama, FAQ (“By default, Ollama uses a context window size of 4096 tokens”,
OLLAMA_CONTEXT_LENGTH,OLLAMA_NUM_PARALLEL= 1,OLLAMA_MAX_LOADED_MODELS= 3 × GPU) and Context length (app defaults: 4k / 32k / 256k depending on VRAM), accessed 2026-10-10. https://docs.ollama.com/faq · https://docs.ollama.com/context-length ↩ ↩2 -
VulnCheck, Ollama 0.30.0 through 0.33.2 SSRF via Cross-Host Tensor Blob Redirect (CVE-2026-85180, CVSS 4.0 8.7, unauthenticated, no fixed version named), 2026-09-03 · Ollama, PR #18512 “x/transfer, server: tighten redirect handling for registry requests” (validation of the redirect target for registry and blob downloads; commit of 2026-09-17, shipped in 0.34.2). https://www.vulncheck.com/advisories/ollama-0.30.0-through-0.33.2-ssrf-via-cross-host-tensor-blob-redirect · https://github.com/ollama/ollama/pull/18512 ↩ ↩2
-
MITRE, CVE-2026-102697 (Ollama 0.14.0 → < 0.31.2, bypass of agent-mode Bash command approval, CVSS v3.1 7.8), published 2026-09-29 · Ollama, Release v0.31.2 (“Hardened GGUF model creation”), 6 July 2026. https://cveawg.mitre.org/api/cve/CVE-2026-102697 · https://github.com/ollama/ollama/releases/tag/v0.31.2 ↩