Skip to content

🔄 Migrate from Ollama to vLLM


When to migrate?

Ollama remains the best choice for solo development and small teams. Migration to vLLM is justified when:

SignalIndicative threshold
Simultaneous usersseveral concurrent requests on the same model (Ollama handles only one at a time by default, OLLAMA_NUM_PARALLEL=1)1
p95 latency> 10 s for an 8B model
Target throughput> 50 tok/s aggregate
Concurrent requests> 20/min at peak
Defined SLATTFT < 2 s guaranteed

API compatibility — what changes, what does not

Both services expose an OpenAI-compatible API on /v1/. In most cases, only the base URL changes.

What stays the same

# Before (Ollama)
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# After (vLLM)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR-TOKEN-TO-REPLACE")
# The code below is identical in both cases
response = client.chat.completions.create(
model="llama3.1", # see "model names" section below
messages=[{"role": "user", "content": "Hello"}],
temperature=0.7,
max_tokens=500
)

What changes

FeatureOllamavLLM
Default port114348000
AuthenticationNone (key ignored)--api-key (does not cover all routes: reverse proxy mandatory, see Security)
Model formatGGUF (native)HuggingFace safetensors, AWQ, GPTQ
Pull model endpointPOST /api/pullNot supported (pre-load)
Generate endpoint (legacy)POST /api/generateNot supported (use /v1/)
StreamSupported (OpenAI wire format since 0.32.6)3Supported
EmbeddingsPOST /api/embed (native, input field)4 or POST /v1/embeddingsPOST /v1/embeddings
Responses APIPOST /v1/responses (since 0.13.3, stateless variant)POST /v1/responses

If your clients were written before August 2026 against Ollama’s streaming, revalidate them in phase 2: 0.32.6 (August 4, 2026) aligned the chunk format with OpenAI (role on the first chunk only, finish_reason on its own chunk, usage via stream_options.include_usage) and truncated responses now return finish_reason: "length"3.


Model names

Ollama uses its own names (llama3.2, qwen2.5:14b). vLLM uses HuggingFace identifiers (meta-llama/Llama-3.2-3B-Instruct), but you can set an alias with --served-model-name to keep compatibility:

Fenêtre de terminal
# vLLM with Ollama-compatible alias
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--served-model-name llama3.1 \ # ← client sends "llama3.1", vLLM understands
--port 8000

Converting GGUF models

vLLM can load a GGUF (vllm-gguf-plugin plugin, e.g. vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B), but as of Q4 2026 the project describes this support as “highly experimental and under-optimized”: it is suitable for validating a model, not for production6. Three options, in order of preference:

Most Ollama models have an official HuggingFace equivalent:

Ollama modelHuggingFace equivalent
llama3.2meta-llama/Llama-3.2-3B-Instruct
llama3.1:70bmeta-llama/Llama-3.1-70B-Instruct
qwen2.5:14bQwen/Qwen2.5-14B-Instruct
qwen2.5-coder:32bQwen/Qwen2.5-Coder-32B-Instruct
phi4microsoft/phi-4
deepseek-r1:70bdeepseek-ai/DeepSeek-R1-Distill-Llama-70B

These equivalences remain valid, but these models belong to the generation that Ollama ≥ 0.32 (July 2026) flags as “legacy” at ollama launch (CodeLlama, Qwen2.5, Llama 3.x, Mistral, base DeepSeek-R1); the current library as of Q4 2026 is qwen3.5 / qwen3.6, gemma4, qwen3-coder, whose Hugging Face weights are found the same way7.

Fenêtre de terminal
# Download via Hugging Face CLI (`hf` command, which replaces `huggingface-cli`)
pip install huggingface_hub
hf auth login # HF token required for gated models (Llama)
hf download meta-llama/Llama-3.1-8B-Instruct \
--local-dir /data/models/llama3.1-8b

Target safetensors repositories exclusively: never accept .bin / pickle weights, even “scanned” ones — malicious pickle loads evade model scanners (ShadowPickle, July 2026)8. The hf command is the documented form of the Hugging Face CLI as of Q4 20269.

Option B — Use a pre-quantized AWQ version

For heavy models (70B+), AWQ versions are lighter and natively handled by vLLM10:

Fenêtre de terminal
# AWQ 4-bit — quality close to BF16 with ~25% of VRAM
vllm serve hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
--quantization awq_marlin \
--dtype half

Option C — Convert GGUF to safetensors (advanced)

If you have a custom GGUF model (fine-tuned, merged), conversion is possible via the Transformers GGUF loader (llama.cpp only does the reverse direction, HF → GGUF, with convert_hf_to_gguf.py)11:

Fenêtre de terminal
pip install transformers gguf
# Convert GGUF → safetensors with Transformers (dequantizes to bf16)
python - <<'PY'
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GgufConfig
repo, f = "/path/to/model-dir", "model.gguf"
m = AutoModelForCausalLM.from_pretrained(
repo, gguf_file=f,
quantization_config=GgufConfig(dequantize=True),
dtype=torch.bfloat16,
)
t = AutoTokenizer.from_pretrained(repo, gguf_file=f)
m.save_pretrained("/path/to/output"); t.save_pretrained("/path/to/output")
PY

The loader dequantizes Llama, Mistral, Qwen2, Phi3, etc.; check the list of supported architectures in the Transformers documentation11.


Zero-downtime cutover strategy

Phase 1 — Parallel deployment

Run vLLM on a different port (8001) alongside Ollama (11434). Do not touch clients yet.

Fenêtre de terminal
# vLLM on port 8001 (staging)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--served-model-name llama3.2 \
--port 8001

Phase 2 — Qualification

Compare results on your real prompts12; vllm bench serve --model … --dataset-name sharegpt additionally measures throughput and TTFT under load.

Fenêtre de terminal
# A/B comparison script
for prompt in "Summarize this contract" "Draft an email" "Analyze this code"; do
echo "=== Ollama ==="
curl -s http://localhost:11434/v1/chat/completions \
-d "{\"model\":\"llama3.2\",\"messages\":[{\"role\":\"user\",\"content\":\"$prompt\"}]}" \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['choices'][0]['message']['content'][:200])"
echo "=== vLLM ==="
curl -s http://localhost:8001/v1/chat/completions \
-H "Authorization: Bearer sk-token" \
-d "{\"model\":\"llama3.2\",\"messages\":[{\"role\":\"user\",\"content\":\"$prompt\"}]}" \
| python3 -c "import sys,json; r=json.load(sys.stdin); print(r['choices'][0]['message']['content'][:200])"
done

Phase 3 — Cutover via reverse proxy

Change only the reverse proxy configuration (Caddy or Nginx), not the clients.

Caddy — switch backend:

# Before
reverse_proxy localhost:11434
# After (change only this line)
reverse_proxy localhost:8000

Hot reload Caddy without downtime:

Fenêtre de terminal
caddy reload --config Caddyfile

Phase 4 — Stop Ollama

After 48 hours with no reported issues:

Fenêtre de terminal
# Stop Ollama
sudo systemctl stop ollama
sudo systemctl disable ollama
# Free memory from Ollama cached models
# (optional, GGUF files remain on disk)

Rollback plan

Fenêtre de terminal
# 1. Restart Ollama
sudo systemctl start ollama
# 2. Point reverse proxy back to Ollama
# Change backend in Caddyfile/nginx.conf → port 11434
caddy reload --config Caddyfile
# 3. Verify
curl http://localhost/v1/models

Full rollback takes < 2 minutes if Ollama was only stopped (not uninstalled).


Migration checklist

□ vLLM ≥ 0.34.0 installed, trust_remote_code disabled
□ HuggingFace equivalent identified for each Ollama model in use
□ Models downloaded and loaded in vLLM (/health test OK)
□ --served-model-name configured for name compatibility
□ Bearer token authentication configured and tested on clients
□ Parallel deployment validated (phases 1–2 complete)
□ Performance compared on domain prompts (throughput, TTFT, quality)
□ Prometheus monitoring active before cutover
□ Reverse proxy reconfigured and reloaded without downtime
□ 48h monitoring period post-cutover
□ Rollback procedure documented and tested

The minimum version is not a detail: vLLM fixed several remote code executions and denials of service in 2026; full hardening is described in Configure vLLM multi-GPU and Local inference security.


See also


Sources and references

Footnotes

  1. Ollama, FAQ — How does Ollama handle concurrent requests? (OLLAMA_NUM_PARALLEL defaults to 1, queueing), accessed 2026-10-10. https://docs.ollama.com/faq ↩

  2. vLLM Project, vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (continuous batching, dynamic KV Cache management), 20 June 2023, verified 2026-10-10. https://vllm.ai/blog/2023-06-20-vllm ↩

  3. Ollama, Release v0.32.6 (“/v1/chat/completions streaming now matches OpenAI’s wire format”; finish_reason: "length" for truncated responses), August 4, 2026. https://github.com/ollama/ollama/releases/tag/v0.32.6 ↩ ↩2

  4. Ollama, API — Generate embeddings (POST /api/embed, input field string or array; no /api/embeddings endpoint documented), accessed 2026-10-10. https://docs.ollama.com/api/embed ↩

  5. vLLM Project, Online Serving — OpenAI-Compatible Server (supported endpoints /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses), accessed 2026-10-10 · Ollama, OpenAI compatibility (/v1/responses since 0.13.3, API key “required but ignored”), accessed 2026-10-10. https://docs.vllm.ai/en/stable/serving/online_serving/ · https://docs.ollama.com/api/openai-compatibility ↩

  6. vLLM Project, Quantization — GGUF (vllm-gguf-plugin plugin, “highly experimental and under-optimized” support, base model tokenizer recommended), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/quantization/gguf/ ↩

  7. Ollama, Release v0.32.0 (deprecation warning for CodeLlama, Qwen2.5(-coder), Llama 3.x, Mistral, StarCoder and base DeepSeek-R1), 11 July 2026 · Ollama, Library (qwen3.5, qwen3.6, gemma4, qwen3-coder), accessed 2026-10-10. https://github.com/ollama/ollama/releases/tag/v0.32.0 · https://ollama.com/library ↩

  8. ShadowPickle: Evading Machine Learning Model Scanners via Stealthy Pickle Deserialization Attacks (arXiv:2607.17503; ten model scanners bypassed), July 2026. https://arxiv.org/abs/2607.17503 ↩

  9. Hugging Face, Command Line Interface (CLI) (hf auth login, hf download … --local-dir), accessed 2026-10-10. https://huggingface.co/docs/huggingface_hub/guides/cli ↩

  10. vLLM Project, Quantization — AWQ (“The AutoAWQ library is deprecated”, workflow taken over by llm-compressor, Marlin kernels), accessed 2026-10-10 · casper-hansen, AutoAWQ (repository archived on 2025-05-11, “officially deprecated”). https://docs.vllm.ai/en/stable/features/quantization/auto_awq/ · https://github.com/casper-hansen/AutoAWQ ↩ ↩2

  11. Hugging Face, Transformers — GGUF (from_pretrained(gguf_file=…), GgufConfig(dequantize=True), supported architectures, export via save_pretrained), accessed 2026-10-09. https://huggingface.co/docs/transformers/gguf ↩ ↩2

  12. vLLM Project, Benchmarking (vllm bench serve, latency, throughput), accessed 2026-10-10. https://docs.vllm.ai/en/stable/benchmarking/ ↩

  13. Ollama, Release v0.40.2 (models “upgraded in the background the first time you run them”, backups kept, re-pull required when rolling back to < 0.40), 8 October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.2 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.