Ollama
π Short definition
A distribution and CLI that runs LLMs locally in a few commands, via llama.cpp (Linux, Windows, GGUF format) or, on Apple Silicon, via Appleβs MLX engine by default, with an OpenAI-compatible API server on port 114341.
π Detailed definition
Ollama is the shortest path to test an LLM on-premise: ollama pull, ollama run, then connect a UI (Open WebUI, Python script, etc.).
Under the hood, Ollama relies on llama.cpp (C/C++) β aggressive quantization (Q4_K_M, etc.), CPU/GPU offloading on modest machines, GGUF format β and, on Apple Silicon, on the MLX engine (safetensors weights, unified memory), used by default since 0.40 for supported architectures2.
π‘ Why it matters for on-prem AI
- Scenario A (dev lab) and first steps of Scenario B (SMB): validate model, prompts, light RAG before switching to vLLM on a GPU server (or staying on llama.cpp / MLX for a Mac or APU appliance).
- Reference for Getting started with Ollama and evaluating a local model guides.
- Local embedding models (
nomic-embed-text, etc.) via the same API.
β οΈ Common pitfalls
- Serving multiple simultaneous users in production: by default Ollama handles only one request at a time per model (
OLLAMA_NUM_PARALLEL=1) and queues the others; raising this parallelism costs memory (β parallelism Γ context) and does not replace the continuous batching of a production engine3. - Pulling a
:cloudmodel believing you stay on-prem: some library models exist only in a hosted version (kimi-k3:cloud,glm-5.2:cloud) and send requests off the machine; the account offered at first launch is only used for these cloud models and private repositories, local inference does not need it (details by version in Getting started with Ollama)4. - Confusing βdownloaded modelβ with βmodel suited to the businessβ: always validate with a golden dataset.
- Exposing API 11434 without authentication on the internal network: see inference security.
π To go deeper
- βοΈ Inference engines β Ollama vs vLLM limits
- π Getting started with Ollama
- π οΈ Scenario A β Dev Lab
π See also
Footnotes
-
Ollama β site and documentation. https://ollama.com/ β©
-
Ollama, Release v0.40.0 (βModels run on MLX on Apple Silicon by defaultβ), October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.0 β©
-
Ollama, FAQ β How does Ollama handle concurrent requests? (
OLLAMA_NUM_PARALLELdefaults to 1,OLLAMA_MAX_LOADED_MODELS, queueing), accessed 2026-10-10. https://docs.ollama.com/faq β see also βοΈ Inference engines. β© -
Ollama, Release v0.32.0 (
ollamawith no argument launches an agent, default entryglm-5.2:cloud), 11 July 2026 Β· Release v0.32.6 (ollama run kimi-k3offerskimi-k3:cloudβfor cloud-only models that publish no default tagβ), 4 August 2026 Β· Release v0.34.2 (first-run screen βsign in or continue locallyβ), 15 September 2026. https://github.com/ollama/ollama/releases/tag/v0.32.0 Β· https://github.com/ollama/ollama/releases/tag/v0.32.6 Β· https://github.com/ollama/ollama/releases/tag/v0.34.2 β©