Skip to content

Ollama

πŸ“ Short definition

A distribution and CLI that runs LLMs locally in a few commands, via llama.cpp (Linux, Windows, GGUF format) or, on Apple Silicon, via Apple’s MLX engine by default, with an OpenAI-compatible API server on port 114341.

πŸ“– Detailed definition

Ollama is the shortest path to test an LLM on-premise: ollama pull, ollama run, then connect a UI (Open WebUI, Python script, etc.).

Under the hood, Ollama relies on llama.cpp (C/C++) β€” aggressive quantization (Q4_K_M, etc.), CPU/GPU offloading on modest machines, GGUF format β€” and, on Apple Silicon, on the MLX engine (safetensors weights, unified memory), used by default since 0.40 for supported architectures2.

πŸ’‘ Why it matters for on-prem AI

  • Scenario A (dev lab) and first steps of Scenario B (SMB): validate model, prompts, light RAG before switching to vLLM on a GPU server (or staying on llama.cpp / MLX for a Mac or APU appliance).
  • Reference for Getting started with Ollama and evaluating a local model guides.
  • Local embedding models (nomic-embed-text, etc.) via the same API.

⚠️ Common pitfalls

  • Serving multiple simultaneous users in production: by default Ollama handles only one request at a time per model (OLLAMA_NUM_PARALLEL=1) and queues the others; raising this parallelism costs memory (∝ parallelism Γ— context) and does not replace the continuous batching of a production engine3.
  • Pulling a :cloud model believing you stay on-prem: some library models exist only in a hosted version (kimi-k3:cloud, glm-5.2:cloud) and send requests off the machine; the account offered at first launch is only used for these cloud models and private repositories, local inference does not need it (details by version in Getting started with Ollama)4.
  • Confusing β€œdownloaded model” with β€œmodel suited to the business”: always validate with a golden dataset.
  • Exposing API 11434 without authentication on the internal network: see inference security.

πŸ“š To go deeper

  1. βš™οΈ Inference engines β€” Ollama vs vLLM limits
  2. πŸš€ Getting started with Ollama
  3. πŸ› οΈ Scenario A β€” Dev Lab

πŸ”— See also

Footnotes

  1. Ollama β€” site and documentation. https://ollama.com/ ↩

  2. Ollama, Release v0.40.0 (β€œModels run on MLX on Apple Silicon by default”), October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.0 ↩

  3. Ollama, FAQ β€” How does Ollama handle concurrent requests? (OLLAMA_NUM_PARALLEL defaults to 1, OLLAMA_MAX_LOADED_MODELS, queueing), accessed 2026-10-10. https://docs.ollama.com/faq β€” see also βš™οΈ Inference engines. ↩

  4. Ollama, Release v0.32.0 (ollama with no argument launches an agent, default entry glm-5.2:cloud), 11 July 2026 Β· Release v0.32.6 (ollama run kimi-k3 offers kimi-k3:cloud β€œfor cloud-only models that publish no default tag”), 4 August 2026 Β· Release v0.34.2 (first-run screen β€œsign in or continue locally”), 15 September 2026. https://github.com/ollama/ollama/releases/tag/v0.32.0 Β· https://github.com/ollama/ollama/releases/tag/v0.32.6 Β· https://github.com/ollama/ollama/releases/tag/v0.34.2 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.