🔄 Migrate from Ollama to vLLM
When to migrate?
Ollama remains the best choice for solo development and small teams. Migration to vLLM is justified when:
| Signal | Indicative threshold |
|---|---|
| Simultaneous users | several concurrent requests on the same model (Ollama handles only one at a time by default, OLLAMA_NUM_PARALLEL=1)1 |
| p95 latency | > 10 s for an 8B model |
| Target throughput | > 50 tok/s aggregate |
| Concurrent requests | > 20/min at peak |
| Defined SLA | TTFT < 2 s guaranteed |
API compatibility — what changes, what does not
Both services expose an OpenAI-compatible API on /v1/. In most cases, only the base URL changes.
What stays the same
# Before (Ollama)client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# After (vLLM)client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR-TOKEN-TO-REPLACE")
# The code below is identical in both casesresponse = client.chat.completions.create( model="llama3.1", # see "model names" section below messages=[{"role": "user", "content": "Hello"}], temperature=0.7, max_tokens=500)What changes
| Feature | Ollama | vLLM |
|---|---|---|
| Default port | 11434 | 8000 |
| Authentication | None (key ignored) | --api-key (does not cover all routes: reverse proxy mandatory, see Security) |
| Model format | GGUF (native) | HuggingFace safetensors, AWQ, GPTQ |
| Pull model endpoint | POST /api/pull | Not supported (pre-load) |
| Generate endpoint (legacy) | POST /api/generate | Not supported (use /v1/) |
| Stream | Supported (OpenAI wire format since 0.32.6)3 | Supported |
| Embeddings | POST /api/embed (native, input field)4 or POST /v1/embeddings | POST /v1/embeddings |
| Responses API | POST /v1/responses (since 0.13.3, stateless variant) | POST /v1/responses |
If your clients were written before August 2026 against Ollama’s streaming, revalidate them in phase 2: 0.32.6 (August 4, 2026) aligned the chunk format with OpenAI (role on the first chunk only, finish_reason on its own chunk, usage via stream_options.include_usage) and truncated responses now return finish_reason: "length"3.
Model names
Ollama uses its own names (llama3.2, qwen2.5:14b). vLLM uses HuggingFace identifiers (meta-llama/Llama-3.2-3B-Instruct), but you can set an alias with --served-model-name to keep compatibility:
# vLLM with Ollama-compatible aliasvllm serve meta-llama/Llama-3.1-8B-Instruct \ --served-model-name llama3.1 \ # ← client sends "llama3.1", vLLM understands --port 8000Converting GGUF models
vLLM can load a GGUF (vllm-gguf-plugin plugin, e.g. vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B), but as of Q4 2026 the project describes this support as “highly experimental and under-optimized”: it is suitable for validating a model, not for production6. Three options, in order of preference:
Option A — Download native HuggingFace weights (recommended)
Most Ollama models have an official HuggingFace equivalent:
| Ollama model | HuggingFace equivalent |
|---|---|
llama3.2 | meta-llama/Llama-3.2-3B-Instruct |
llama3.1:70b | meta-llama/Llama-3.1-70B-Instruct |
qwen2.5:14b | Qwen/Qwen2.5-14B-Instruct |
qwen2.5-coder:32b | Qwen/Qwen2.5-Coder-32B-Instruct |
phi4 | microsoft/phi-4 |
deepseek-r1:70b | deepseek-ai/DeepSeek-R1-Distill-Llama-70B |
These equivalences remain valid, but these models belong to the generation that Ollama ≥ 0.32 (July 2026) flags as “legacy” at ollama launch (CodeLlama, Qwen2.5, Llama 3.x, Mistral, base DeepSeek-R1); the current library as of Q4 2026 is qwen3.5 / qwen3.6, gemma4, qwen3-coder, whose Hugging Face weights are found the same way7.
# Download via Hugging Face CLI (`hf` command, which replaces `huggingface-cli`)pip install huggingface_hubhf auth login # HF token required for gated models (Llama)
hf download meta-llama/Llama-3.1-8B-Instruct \ --local-dir /data/models/llama3.1-8bTarget safetensors repositories exclusively: never accept .bin / pickle weights, even “scanned” ones — malicious pickle loads evade model scanners (ShadowPickle, July 2026)8. The hf command is the documented form of the Hugging Face CLI as of Q4 20269.
Option B — Use a pre-quantized AWQ version
For heavy models (70B+), AWQ versions are lighter and natively handled by vLLM10:
# AWQ 4-bit — quality close to BF16 with ~25% of VRAMvllm serve hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \ --quantization awq_marlin \ --dtype halfOption C — Convert GGUF to safetensors (advanced)
If you have a custom GGUF model (fine-tuned, merged), conversion is possible via the Transformers GGUF loader (llama.cpp only does the reverse direction, HF → GGUF, with convert_hf_to_gguf.py)11:
pip install transformers gguf
# Convert GGUF → safetensors with Transformers (dequantizes to bf16)python - <<'PY'import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, GgufConfigrepo, f = "/path/to/model-dir", "model.gguf"m = AutoModelForCausalLM.from_pretrained( repo, gguf_file=f, quantization_config=GgufConfig(dequantize=True), dtype=torch.bfloat16,)t = AutoTokenizer.from_pretrained(repo, gguf_file=f)m.save_pretrained("/path/to/output"); t.save_pretrained("/path/to/output")PYThe loader dequantizes Llama, Mistral, Qwen2, Phi3, etc.; check the list of supported architectures in the Transformers documentation11.
Zero-downtime cutover strategy
Phase 1 — Parallel deployment
Run vLLM on a different port (8001) alongside Ollama (11434). Do not touch clients yet.
# vLLM on port 8001 (staging)vllm serve meta-llama/Llama-3.1-8B-Instruct \ --served-model-name llama3.2 \ --port 8001Phase 2 — Qualification
Compare results on your real prompts12; vllm bench serve --model … --dataset-name sharegpt additionally measures throughput and TTFT under load.
# A/B comparison scriptfor prompt in "Summarize this contract" "Draft an email" "Analyze this code"; do echo "=== Ollama ===" curl -s http://localhost:11434/v1/chat/completions \ -d "{\"model\":\"llama3.2\",\"messages\":[{\"role\":\"user\",\"content\":\"$prompt\"}]}" \ | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['choices'][0]['message']['content'][:200])"
echo "=== vLLM ===" curl -s http://localhost:8001/v1/chat/completions \ -H "Authorization: Bearer sk-token" \ -d "{\"model\":\"llama3.2\",\"messages\":[{\"role\":\"user\",\"content\":\"$prompt\"}]}" \ | python3 -c "import sys,json; r=json.load(sys.stdin); print(r['choices'][0]['message']['content'][:200])"donePhase 3 — Cutover via reverse proxy
Change only the reverse proxy configuration (Caddy or Nginx), not the clients.
Caddy — switch backend:
# Beforereverse_proxy localhost:11434
# After (change only this line)reverse_proxy localhost:8000Hot reload Caddy without downtime:
caddy reload --config CaddyfilePhase 4 — Stop Ollama
After 48 hours with no reported issues:
# Stop Ollamasudo systemctl stop ollamasudo systemctl disable ollama
# Free memory from Ollama cached models# (optional, GGUF files remain on disk)Rollback plan
# 1. Restart Ollamasudo systemctl start ollama
# 2. Point reverse proxy back to Ollama# Change backend in Caddyfile/nginx.conf → port 11434caddy reload --config Caddyfile
# 3. Verifycurl http://localhost/v1/modelsFull rollback takes < 2 minutes if Ollama was only stopped (not uninstalled).
Migration checklist
□ vLLM ≥ 0.34.0 installed, trust_remote_code disabled□ HuggingFace equivalent identified for each Ollama model in use□ Models downloaded and loaded in vLLM (/health test OK)□ --served-model-name configured for name compatibility□ Bearer token authentication configured and tested on clients□ Parallel deployment validated (phases 1–2 complete)□ Performance compared on domain prompts (throughput, TTFT, quality)□ Prometheus monitoring active before cutover□ Reverse proxy reconfigured and reloaded without downtime□ 48h monitoring period post-cutover□ Rollback procedure documented and testedThe minimum version is not a detail: vLLM fixed several remote code executions and denials of service in 2026; full hardening is described in Configure vLLM multi-GPU and Local inference security.
See also
- 🚀 Getting started with Ollama — if you roll back or test in parallel
- ⚙️ Configure vLLM multi-GPU — full configuration of the new backend
- 📊 Prometheus + Grafana monitoring — essential before production cutover
- 🔒 Local inference security — authentication and reverse proxy
Sources and references
Footnotes
-
Ollama, FAQ — How does Ollama handle concurrent requests? (
OLLAMA_NUM_PARALLELdefaults to 1, queueing), accessed 2026-10-10. https://docs.ollama.com/faq ↩ -
vLLM Project, vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (continuous batching, dynamic KV Cache management), 20 June 2023, verified 2026-10-10. https://vllm.ai/blog/2023-06-20-vllm ↩
-
Ollama, Release v0.32.6 (“
/v1/chat/completionsstreaming now matches OpenAI’s wire format”;finish_reason: "length"for truncated responses), August 4, 2026. https://github.com/ollama/ollama/releases/tag/v0.32.6 ↩ ↩2 -
Ollama, API — Generate embeddings (
POST /api/embed,inputfield string or array; no/api/embeddingsendpoint documented), accessed 2026-10-10. https://docs.ollama.com/api/embed ↩ -
vLLM Project, Online Serving — OpenAI-Compatible Server (supported endpoints
/v1/chat/completions,/v1/completions,/v1/embeddings,/v1/responses), accessed 2026-10-10 · Ollama, OpenAI compatibility (/v1/responsessince 0.13.3, API key “required but ignored”), accessed 2026-10-10. https://docs.vllm.ai/en/stable/serving/online_serving/ · https://docs.ollama.com/api/openai-compatibility ↩ -
vLLM Project, Quantization — GGUF (
vllm-gguf-pluginplugin, “highly experimental and under-optimized” support, base model tokenizer recommended), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/quantization/gguf/ ↩ -
Ollama, Release v0.32.0 (deprecation warning for CodeLlama, Qwen2.5(-coder), Llama 3.x, Mistral, StarCoder and base DeepSeek-R1), 11 July 2026 · Ollama, Library (
qwen3.5,qwen3.6,gemma4,qwen3-coder), accessed 2026-10-10. https://github.com/ollama/ollama/releases/tag/v0.32.0 · https://ollama.com/library ↩ -
ShadowPickle: Evading Machine Learning Model Scanners via Stealthy Pickle Deserialization Attacks (arXiv:2607.17503; ten model scanners bypassed), July 2026. https://arxiv.org/abs/2607.17503 ↩
-
Hugging Face, Command Line Interface (CLI) (
hf auth login,hf download … --local-dir), accessed 2026-10-10. https://huggingface.co/docs/huggingface_hub/guides/cli ↩ -
vLLM Project, Quantization — AWQ (“The AutoAWQ library is deprecated”, workflow taken over by
llm-compressor, Marlin kernels), accessed 2026-10-10 · casper-hansen, AutoAWQ (repository archived on 2025-05-11, “officially deprecated”). https://docs.vllm.ai/en/stable/features/quantization/auto_awq/ · https://github.com/casper-hansen/AutoAWQ ↩ ↩2 -
Hugging Face, Transformers — GGUF (
from_pretrained(gguf_file=…),GgufConfig(dequantize=True), supported architectures, export viasave_pretrained), accessed 2026-10-09. https://huggingface.co/docs/transformers/gguf ↩ ↩2 -
vLLM Project, Benchmarking (
vllm bench serve,latency,throughput), accessed 2026-10-10. https://docs.vllm.ai/en/stable/benchmarking/ ↩ -
Ollama, Release v0.40.2 (models “upgraded in the background the first time you run them”, backups kept, re-pull required when rolling back to < 0.40), 8 October 2026. https://github.com/ollama/ollama/releases/tag/v0.40.2 ↩