⚙️ Configure vLLM for multi-GPU production
1. Installation
Via pip (recommended)
# Python 3.11-3.14; default wheel CUDA 13.0# (since vLLM 0.33: CUDA 13.0 only, no more CUDA 12.9 variant)pip install vllm
# Verificationpython -c "import vllm; print(vllm.__version__)"Via Docker (recommended for production)
The official image avoids CUDA dependency conflicts1:
docker pull vllm/vllm-openai:latest
docker run --runtime nvidia --gpus all \ -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model meta-llama/Llama-3.1-8B-Instruct \ --dtype auto2. Single-GPU configuration
Minimal startup
vllm serve meta-llama/Llama-3.1-8B-Instruct \ --host 0.0.0.0 \ --port 8000Essential parameters
vllm serve meta-llama/Llama-3.1-70B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --dtype bfloat16 \ # native bf16 on Ampere+, more stable than fp16 --max-model-len 8192 \ # max context window (limits KV Cache) --gpu-memory-utilization 1.0 \ # fraction of VRAM allocated to vLLM (default 1.0 since 0.34; omit on a dedicated GPU) --max-num-seqs 256 \ # max concurrent requests in continuous batching --served-model-name llama-70b # API alias (avoids exposing HF path)Critical parameter — gpu-memory-utilization:
vLLM reserves the indicated fraction of VRAM for the whole engine (weights, activations and KV Cache). Since 0.34, the extensible KV cache is enabled by default: vLLM measures free memory after CUDA graph capture, and gpu-memory-utilization defaults to 1.0 (previously 0.92). On a dedicated GPU, no longer set a value: an explicit value such as 0.92 now yields a smaller cache. Set a value (0.85-0.9) only if other processes share the GPU or memory (DGX Spark); --no-enable-extensible-kv-cache restores the previous sizing34.
Quantized models (AWQ / GPTQ / FP8)
# AWQ model (better quality per memory vs GGUF Q4)vllm serve TheBloke/Llama-2-70B-Chat-AWQ \ --quantization awq \ --dtype auto
# GPTQ modelvllm serve TheBloke/Llama-2-70B-GPTQ \ --quantization gptq \ --dtype float16
# Online FP8, without a pre-quantized checkpoint (Ada Lovelace / Hopper / Blackwell)# vLLM ≥ 0.31: the method is called fp8_per_tensor; the former "fp8" name redirects[^13]vllm serve meta-llama/Llama-3.1-70B-Instruct \ --quantization fp8_per_tensor3. Multi-GPU configuration (Tensor Parallelism)
Tensor Parallelism splits weights of the same layer across several GPUs on one server via NVLink or PCIe. It is the recommended mode for models that do not fit on a single card6.
# 2 GPUs — 70B model in 2 × 40 GBvllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 2 \ --dtype bfloat16
# 4 GPUs — 70B model with comfortable headroomvllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 4
# 8 GPUs — 405B model or large MoEvllm serve meta-llama/Llama-3.1-405B-Instruct \ --tensor-parallel-size 8 \ --pipeline-parallel-size 1 \ --dtype bfloat16Sizing rule:
tensor-parallel-sizemust divide the model’s number of attention heads (in practice 1, 2, 4 or 8 for common models)6- Each GPU needs
model_size / tensor_parallel_sizeVRAM - Interconnect drives efficiency: NVLink >> PCIe (see Multi-GPU stations)
graph LR
A[API Request] --> B[vLLM Scheduler]
B --> C[GPU 0 — 1/4 of the weights of each layer 0-79]
B --> D[GPU 1 — 1/4 of the weights of each layer 0-79]
B --> E[GPU 2 — 1/4 of the weights of each layer 0-79]
B --> F[GPU 3 — 1/4 of the weights of each layer 0-79]
C & D & E & F -->|all-reduce at each layer (NVLink or PCIe)| G[Response]
4. Multi-node deployment with Ray
To go beyond one server’s capacity, vLLM uses Ray to distribute the model across machines7.
Network prerequisites
Nodes must see each other on a low-latency network. Ideally RoCE/InfiniBand: the vLLM docs recommend enabling InfiniBand (--privileged -e NCCL_IB_HCA=mlx5 in run_cluster.sh) and checking with NCCL_DEBUG=TRACE that NCCL uses NET/IB/GDRDMA and not NET/Socket7. 25 Gb Ethernet remains usable for Pipeline Parallelism (less bandwidth-demanding than Tensor Parallelism), at the cost of a higher TTFT — the vLLM docs give no numeric threshold, measure on your workload.
Ray cluster configuration
# === On HEAD node (node 0) ===pip install ray vllm
# Start Ray head processray start --head --port=6379
# === On each WORKER node (nodes 1, 2, ...) ===pip install ray vllm
# Join cluster (replace HEAD_IP with head node IP)ray start --address='HEAD_IP:6379'
# === Verify cluster ===ray status# → shows connected nodes and available GPUsLaunch vLLM on the cluster
# On HEAD node — vLLM uses Ray to distribute automaticallyvllm serve meta-llama/Llama-3.1-405B-Instruct \ --tensor-parallel-size 4 \ # 4 GPUs per node --pipeline-parallel-size 2 \ # 2 nodes --distributed-executor-backend ray \ --host 0.0.0.0 \ --port 8000vLLM and Ray handle distribution automatically: the first 4 GPUs (node 0) run early layers, the next 4 (node 1) run the rest7. The official docs provide examples/ray_serving/run_cluster.sh to start head and workers in the vllm/vllm-openai image (one VLLM_HOST_IP per node); without Ray, run vllm serve … --nnodes 2 --node-rank 0 --master-addr HEAD_IP on the head and --node-rank 1 --headless on the worker7.
Prefill / Decode disaggregation
Advanced (experimental) architecture that isolates nodes dedicated to Prefill (prompt read, compute-bound) and others to Decode (generation, memory-bandwidth-bound)8. The TTFT gain depends on the KV connector and prompt length — the vLLM docs publish no figure: measure it with vllm bench serve8.
# P/D disaggregation — same flag on both roles, NIXL connector# (kv_role: "kv_producer" on the Prefill side, "kv_consumer" on the Decode side, "kv_both" for both)vllm serve ... \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'The connectors available as of Q4 2026 are NixlConnector, LMCacheConnectorV1, MooncakeConnector, OffloadingConnector (CPU offload) and MultiConnector; the feature is documented as “experimental and subject to change”8. The --role and --num-speculative-tokens flags do not exist in vllm serve (speculation is configured via --speculative-config)9.
5. Production configuration — advanced parameters
API authentication
vllm serve ... \ --api-key "YOUR-TOKEN-TO-REPLACE"Or via environment variable:
export VLLM_API_KEY="YOUR-TOKEN-TO-REPLACE"vllm serve ...Clients must send Authorization: Bearer YOUR-TOKEN-TO-REPLACE. Caution: --api-key does not cover all routes: in production, put an authenticating reverse proxy in front of all paths (see Security). Since vLLM 0.32.0 (18 October 2026), --allowed-origins defaults to [] instead of ["*"] (CVE-2026-115917). A web interface that calls vLLM directly from the browser must declare its origin (--allowed-origins '["https://chat.interne.example"]'). Server-side clients (SDKs, curl, LiteLLM) are not affected10.
Limits and timeouts
vllm serve ... \ --max-num-seqs 512 \ # sequences processed per iteration (continuous batching) --max-num-queued-reqs 1024 # max in-flight requests; beyond: HTTP 503 (vLLM ≥ 0.29)# Request logs are disabled by default (--enable-log-requests for debugging)[^8]--max-num-seqs bounds the number of sequences processed per iteration, not the queue (unbounded by default): it is --max-num-queued-reqs (vLLM 0.29) that produces the 503, to be sized around data_parallel_size × max_num_seqs plus the desired queue depth39. vLLM has no per-request timeout on the engine side: set it in the reverse proxy (Caddy reverse_proxy … { transport http { response_header_timeout 120s } }, Nginx proxy_read_timeout 120s) or on the client side9.
KV Cache optimization — FP8 quantization
On NVIDIA GPUs (CUDA 11.8+) and AMD ROCm, FP8 KV Cache quantization halves its footprint compared to BF16; without calibration, all scales are 1.0 — the docs recommend calibrating with llm-compressor and excluding sliding-window layers (--kv-cache-dtype-skip-layers sliding_window)11:
vllm serve ... \ --kv-cache-dtype fp8The KV Cache size actually allocated is written in the startup logs (“GPU KV cache size: … tokens”); there is no dedicated flag to display it9.
Automatic Prefix Caching (APC) — essential for RAG and agents
In multi-user or multi-agent setups, several requests often share the same System Prompt (500–2000 tokens) or the same RAG document in context. Without APC, vLLM computes and stores that prefix KV Cache N times — once per request — wasting VRAM and compute. Since the V1 architecture, APC is enabled by default: there is nothing to do to benefit from it12.
vllm serve meta-llama/Llama-3.1-70B-Instruct \ --max-model-len 8192# APC is on by default; --no-enable-prefix-caching to turn it off,# --prefix-caching-hash-algo xxhash for faster hashing (sha256 by default)How it works: vLLM hashes each KV Cache block (size --block-size, 64 tokens by default since 0.34 instead of 16; only full blocks are reused, so a short shared prefix may no longer benefit from the cache: compare with --block-size 16)3. If a new request starts with the same block sequence as a prior request still in GPU cache, Key/Value vectors are reused directly — without recomputing Prefill12.
Measured impact:
| Scenario | Without APC | With APC |
|---|---|---|
| 20 parallel agents, same 800-token system prompt | TTFT 3–8s each | TTFT < 100ms from 2nd request |
| RAG pipeline: shared document context | full recompute × N | cache hit: ~96% VRAM saved on prefix |
Systemd service (Linux)
[Unit]Description=vLLM Inference ServerAfter=network.target
[Service]Type=simpleUser=vllmEnvironment="HF_HOME=/data/models"Environment="CUDA_VISIBLE_DEVICES=0,1,2,3"# `python -m vllm.entrypoints.openai.api_server` is deprecated since vLLM 0.29: use `vllm serve`# (adapt the `vllm` path to the virtual environment; the model is the positional argument)ExecStart=/usr/local/bin/vllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 4 \ --host 127.0.0.1 \ --port 8000Restart=alwaysRestartSec=10
[Install]WantedBy=multi-user.targetOn ROCm, CUDA_VISIBLE_DEVICES is deprecated since vLLM 0.24 (the engine no longer sets this variable itself); use the device_ids argument. Nothing changes for NVIDIA13.
sudo systemctl daemon-reloadsudo systemctl enable --now vllmsudo journalctl -u vllm -f # follow logs6. Verification and tests
# Health checkcurl http://localhost:8000/health
# List loaded modelscurl http://localhost:8000/v1/models | python3 -m json.tool
# Generation testcurl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR-TOKEN-TO-REPLACE" \ -d '{ "model": "llama-70b", "messages": [{"role": "user", "content": "Hello, are you working?"}], "max_tokens": 100 }'
# Prometheus metricscurl http://localhost:8000/metrics | grep vllmNext steps
- Monitoring → 📊 Prometheus + Grafana monitoring
- Migration from Ollama → 🔄 Migrate from Ollama to vLLM
- Hardening security → 🔒 Local inference security
Sources and references
Footnotes
-
vLLM Project, Installation — GPU (official image
vllm/vllm-openai, Python 3.11–3.14, compute capability ≥ 7.5, default CUDA 13.0 wheel andcu129variant), accessed 2026-10-09. https://docs.vllm.ai/en/stable/getting_started/installation/gpu/ ↩ ↩2 -
vLLM Project, Release v0.28.0 (PyPI wheel and Docker image defaulting to CUDA 13.0,
-cu129variants), 2026-08-26 · Release v0.33.0 (CUDA 13.0 only wheels and images,-cu129variants removed), 2026-11-02. https://github.com/vllm-project/vllm/releases/tag/v0.28.0 · https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3 ↩ -
vLLM Project, Engine Arguments (
--gpu-memory-utilization,--max-model-len,--max-num-seqs,--max-num-queued-reqs, KV Cache behavior), accessed 2026-10-09. https://docs.vllm.ai/en/stable/configuration/engine_args/ ↩ ↩2 ↩3 -
vLLM Project, Release v0.34.0 (“Before you upgrade”: Model Runner V1 removed;
--block-size64 by default instead of 16; extensible KV cache enabled by default andgpu_memory_utilizationat 1.0, previous sizing via--gpu-memory-utilization 0.9or--no-enable-extensible-kv-cache; ROCm 7.2 artifacts removed), 16 November 2026. https://github.com/vllm-project/vllm/releases/tag/v0.34.0 ↩ -
vLLM Project, GGUF (
vllm-gguf-pluginplugin, “highly experimental and under-optimized” support, base model tokenizer), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/quantization/gguf/ ↩ -
vLLM Project, Parallelism and Scaling — Tensor Parallelism (
--tensor-parallel-size, weight sharding, NVLink recommendations). https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ ↩ ↩2 -
vLLM Project, Parallelism and Scaling — Multi-node deployment (
run_cluster.sh,--distributed-executor-backend ray,--nnodes/--node-rank/--headless, InfiniBand and NCCL), accessed 2026-10-09. https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ ↩ ↩2 ↩3 ↩4 -
vLLM Project, Disaggregated Prefill and Decode (
--kv-transfer-config, NIXL / LMCache / Mooncake / Offloading connectors, experimental status), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/disagg_prefill/ ↩ ↩2 ↩3 ↩4 -
vLLM Project, CLI Reference —
vllm serve(exhaustive flag list: no--role,--request-timeout,--calculate-kv-cache-size;--speculative-config), accessed 2026-10-09 · vLLM Project, Release v0.29.0 (addition of--max-num-queued-reqs), 2026-09-09. https://docs.vllm.ai/en/stable/cli/serve/ · https://github.com/vllm-project/vllm/releases/tag/v0.29.0 ↩ ↩2 ↩3 ↩4 -
vLLM Project, Release v0.32.0 (“Default CORS settings no longer allow every origin in the Python and Rust frontends; set
--allowed-originsto permit browser origins”, #61009; empty list by default; fixes CVE-2026-115917, vLLM ≤ 0.31.x), 18 October 2026. https://github.com/vllm-project/vllm/releases/tag/v0.32.0 · https://github.com/vllm-project/vllm/pull/61009 ↩ -
vLLM Project, Quantized KV Cache (
--kv-cache-dtype fp8/fp8_e5m2, CUDA 11.8+ and ROCm,llm-compressorcalibration,--kv-cache-dtype-skip-layers), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache/ ↩ -
vLLM Project, Automatic Prefix Caching (hashing by KV Cache blocks, TTFT impact, tensor parallelism compatibility), accessed 2026-10-10. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ ↩ ↩2 ↩3
-
vLLM Project, Release v0.24.0 (vLLM no longer sets
CUDA_VISIBLE_DEVICES;device_idsargument; variable deprecated on ROCm), 2026-06-29. https://github.com/vllm-project/vllm/releases/tag/v0.24.0 ↩