Skip to content

⚙️ Configure vLLM for multi-GPU production


1. Installation

Fenêtre de terminal
# Python 3.11-3.14; default wheel CUDA 13.0
# (since vLLM 0.33: CUDA 13.0 only, no more CUDA 12.9 variant)
pip install vllm
# Verification
python -c "import vllm; print(vllm.__version__)"

The official image avoids CUDA dependency conflicts1:

Fenêtre de terminal
docker pull vllm/vllm-openai:latest
docker run --runtime nvidia --gpus all \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.1-8B-Instruct \
--dtype auto

2. Single-GPU configuration

Minimal startup

Fenêtre de terminal
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000

Essential parameters

Fenêtre de terminal
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype bfloat16 \ # native bf16 on Ampere+, more stable than fp16
--max-model-len 8192 \ # max context window (limits KV Cache)
--gpu-memory-utilization 1.0 \ # fraction of VRAM allocated to vLLM (default 1.0 since 0.34; omit on a dedicated GPU)
--max-num-seqs 256 \ # max concurrent requests in continuous batching
--served-model-name llama-70b # API alias (avoids exposing HF path)

Critical parameter — gpu-memory-utilization: vLLM reserves the indicated fraction of VRAM for the whole engine (weights, activations and KV Cache). Since 0.34, the extensible KV cache is enabled by default: vLLM measures free memory after CUDA graph capture, and gpu-memory-utilization defaults to 1.0 (previously 0.92). On a dedicated GPU, no longer set a value: an explicit value such as 0.92 now yields a smaller cache. Set a value (0.85-0.9) only if other processes share the GPU or memory (DGX Spark); --no-enable-extensible-kv-cache restores the previous sizing34.

Quantized models (AWQ / GPTQ / FP8)

Fenêtre de terminal
# AWQ model (better quality per memory vs GGUF Q4)
vllm serve TheBloke/Llama-2-70B-Chat-AWQ \
--quantization awq \
--dtype auto
# GPTQ model
vllm serve TheBloke/Llama-2-70B-GPTQ \
--quantization gptq \
--dtype float16
# Online FP8, without a pre-quantized checkpoint (Ada Lovelace / Hopper / Blackwell)
# vLLM ≥ 0.31: the method is called fp8_per_tensor; the former "fp8" name redirects[^13]
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--quantization fp8_per_tensor

3. Multi-GPU configuration (Tensor Parallelism)

Tensor Parallelism splits weights of the same layer across several GPUs on one server via NVLink or PCIe. It is the recommended mode for models that do not fit on a single card6.

Fenêtre de terminal
# 2 GPUs — 70B model in 2 × 40 GB
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 2 \
--dtype bfloat16
# 4 GPUs — 70B model with comfortable headroom
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4
# 8 GPUs — 405B model or large MoE
vllm serve meta-llama/Llama-3.1-405B-Instruct \
--tensor-parallel-size 8 \
--pipeline-parallel-size 1 \
--dtype bfloat16

Sizing rule:

  • tensor-parallel-size must divide the model’s number of attention heads (in practice 1, 2, 4 or 8 for common models)6
  • Each GPU needs model_size / tensor_parallel_size VRAM
  • Interconnect drives efficiency: NVLink >> PCIe (see Multi-GPU stations)
graph LR
    A[API Request] --> B[vLLM Scheduler]
    B --> C[GPU 0 — 1/4 of the weights of each layer 0-79]
    B --> D[GPU 1 — 1/4 of the weights of each layer 0-79]
    B --> E[GPU 2 — 1/4 of the weights of each layer 0-79]
    B --> F[GPU 3 — 1/4 of the weights of each layer 0-79]
    C & D & E & F -->|all-reduce at each layer (NVLink or PCIe)| G[Response]

4. Multi-node deployment with Ray

To go beyond one server’s capacity, vLLM uses Ray to distribute the model across machines7.

Network prerequisites

Nodes must see each other on a low-latency network. Ideally RoCE/InfiniBand: the vLLM docs recommend enabling InfiniBand (--privileged -e NCCL_IB_HCA=mlx5 in run_cluster.sh) and checking with NCCL_DEBUG=TRACE that NCCL uses NET/IB/GDRDMA and not NET/Socket7. 25 Gb Ethernet remains usable for Pipeline Parallelism (less bandwidth-demanding than Tensor Parallelism), at the cost of a higher TTFT — the vLLM docs give no numeric threshold, measure on your workload.

Ray cluster configuration

Fenêtre de terminal
# === On HEAD node (node 0) ===
pip install ray vllm
# Start Ray head process
ray start --head --port=6379
# === On each WORKER node (nodes 1, 2, ...) ===
pip install ray vllm
# Join cluster (replace HEAD_IP with head node IP)
ray start --address='HEAD_IP:6379'
# === Verify cluster ===
ray status
# → shows connected nodes and available GPUs

Launch vLLM on the cluster

Fenêtre de terminal
# On HEAD node — vLLM uses Ray to distribute automatically
vllm serve meta-llama/Llama-3.1-405B-Instruct \
--tensor-parallel-size 4 \ # 4 GPUs per node
--pipeline-parallel-size 2 \ # 2 nodes
--distributed-executor-backend ray \
--host 0.0.0.0 \
--port 8000

vLLM and Ray handle distribution automatically: the first 4 GPUs (node 0) run early layers, the next 4 (node 1) run the rest7. The official docs provide examples/ray_serving/run_cluster.sh to start head and workers in the vllm/vllm-openai image (one VLLM_HOST_IP per node); without Ray, run vllm serve … --nnodes 2 --node-rank 0 --master-addr HEAD_IP on the head and --node-rank 1 --headless on the worker7.

Prefill / Decode disaggregation

Advanced (experimental) architecture that isolates nodes dedicated to Prefill (prompt read, compute-bound) and others to Decode (generation, memory-bandwidth-bound)8. The TTFT gain depends on the KV connector and prompt length — the vLLM docs publish no figure: measure it with vllm bench serve8.

Fenêtre de terminal
# P/D disaggregation — same flag on both roles, NIXL connector
# (kv_role: "kv_producer" on the Prefill side, "kv_consumer" on the Decode side, "kv_both" for both)
vllm serve ... \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

The connectors available as of Q4 2026 are NixlConnector, LMCacheConnectorV1, MooncakeConnector, OffloadingConnector (CPU offload) and MultiConnector; the feature is documented as “experimental and subject to change”8. The --role and --num-speculative-tokens flags do not exist in vllm serve (speculation is configured via --speculative-config)9.


5. Production configuration — advanced parameters

API authentication

Fenêtre de terminal
vllm serve ... \
--api-key "YOUR-TOKEN-TO-REPLACE"

Or via environment variable:

Fenêtre de terminal
export VLLM_API_KEY="YOUR-TOKEN-TO-REPLACE"
vllm serve ...

Clients must send Authorization: Bearer YOUR-TOKEN-TO-REPLACE. Caution: --api-key does not cover all routes: in production, put an authenticating reverse proxy in front of all paths (see Security). Since vLLM 0.32.0 (18 October 2026), --allowed-origins defaults to [] instead of ["*"] (CVE-2026-115917). A web interface that calls vLLM directly from the browser must declare its origin (--allowed-origins '["https://chat.interne.example"]'). Server-side clients (SDKs, curl, LiteLLM) are not affected10.

Limits and timeouts

Fenêtre de terminal
vllm serve ... \
--max-num-seqs 512 \ # sequences processed per iteration (continuous batching)
--max-num-queued-reqs 1024 # max in-flight requests; beyond: HTTP 503 (vLLM ≥ 0.29)
# Request logs are disabled by default (--enable-log-requests for debugging)[^8]

--max-num-seqs bounds the number of sequences processed per iteration, not the queue (unbounded by default): it is --max-num-queued-reqs (vLLM 0.29) that produces the 503, to be sized around data_parallel_size × max_num_seqs plus the desired queue depth39. vLLM has no per-request timeout on the engine side: set it in the reverse proxy (Caddy reverse_proxy … { transport http { response_header_timeout 120s } }, Nginx proxy_read_timeout 120s) or on the client side9.

KV Cache optimization — FP8 quantization

On NVIDIA GPUs (CUDA 11.8+) and AMD ROCm, FP8 KV Cache quantization halves its footprint compared to BF16; without calibration, all scales are 1.0 — the docs recommend calibrating with llm-compressor and excluding sliding-window layers (--kv-cache-dtype-skip-layers sliding_window)11:

Fenêtre de terminal
vllm serve ... \
--kv-cache-dtype fp8

The KV Cache size actually allocated is written in the startup logs (“GPU KV cache size: … tokens”); there is no dedicated flag to display it9.

Automatic Prefix Caching (APC) — essential for RAG and agents

In multi-user or multi-agent setups, several requests often share the same System Prompt (500–2000 tokens) or the same RAG document in context. Without APC, vLLM computes and stores that prefix KV Cache N times — once per request — wasting VRAM and compute. Since the V1 architecture, APC is enabled by default: there is nothing to do to benefit from it12.

Fenêtre de terminal
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--max-model-len 8192
# APC is on by default; --no-enable-prefix-caching to turn it off,
# --prefix-caching-hash-algo xxhash for faster hashing (sha256 by default)

How it works: vLLM hashes each KV Cache block (size --block-size, 64 tokens by default since 0.34 instead of 16; only full blocks are reused, so a short shared prefix may no longer benefit from the cache: compare with --block-size 16)3. If a new request starts with the same block sequence as a prior request still in GPU cache, Key/Value vectors are reused directly — without recomputing Prefill12.

Measured impact:

ScenarioWithout APCWith APC
20 parallel agents, same 800-token system promptTTFT 3–8s eachTTFT < 100ms from 2nd request
RAG pipeline: shared document contextfull recompute × Ncache hit: ~96% VRAM saved on prefix

Systemd service (Linux)

/etc/systemd/system/vllm.service
[Unit]
Description=vLLM Inference Server
After=network.target
[Service]
Type=simple
User=vllm
Environment="HF_HOME=/data/models"
Environment="CUDA_VISIBLE_DEVICES=0,1,2,3"
# `python -m vllm.entrypoints.openai.api_server` is deprecated since vLLM 0.29: use `vllm serve`
# (adapt the `vllm` path to the virtual environment; the model is the positional argument)
ExecStart=/usr/local/bin/vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--host 127.0.0.1 \
--port 8000
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target

On ROCm, CUDA_VISIBLE_DEVICES is deprecated since vLLM 0.24 (the engine no longer sets this variable itself); use the device_ids argument. Nothing changes for NVIDIA13.

Fenêtre de terminal
sudo systemctl daemon-reload
sudo systemctl enable --now vllm
sudo journalctl -u vllm -f # follow logs

6. Verification and tests

Fenêtre de terminal
# Health check
curl http://localhost:8000/health
# List loaded models
curl http://localhost:8000/v1/models | python3 -m json.tool
# Generation test
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR-TOKEN-TO-REPLACE" \
-d '{
"model": "llama-70b",
"messages": [{"role": "user", "content": "Hello, are you working?"}],
"max_tokens": 100
}'
# Prometheus metrics
curl http://localhost:8000/metrics | grep vllm

Next steps


Sources and references

Footnotes

  1. vLLM Project, Installation — GPU (official image vllm/vllm-openai, Python 3.11–3.14, compute capability ≥ 7.5, default CUDA 13.0 wheel and cu129 variant), accessed 2026-10-09. https://docs.vllm.ai/en/stable/getting_started/installation/gpu/ ↩ ↩2

  2. vLLM Project, Release v0.28.0 (PyPI wheel and Docker image defaulting to CUDA 13.0, -cu129 variants), 2026-08-26 · Release v0.33.0 (CUDA 13.0 only wheels and images, -cu129 variants removed), 2026-11-02. https://github.com/vllm-project/vllm/releases/tag/v0.28.0 · https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3 ↩

  3. vLLM Project, Engine Arguments (--gpu-memory-utilization, --max-model-len, --max-num-seqs, --max-num-queued-reqs, KV Cache behavior), accessed 2026-10-09. https://docs.vllm.ai/en/stable/configuration/engine_args/ ↩ ↩2 ↩3

  4. vLLM Project, Release v0.34.0 (“Before you upgrade”: Model Runner V1 removed; --block-size 64 by default instead of 16; extensible KV cache enabled by default and gpu_memory_utilization at 1.0, previous sizing via --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache; ROCm 7.2 artifacts removed), 16 November 2026. https://github.com/vllm-project/vllm/releases/tag/v0.34.0 ↩

  5. vLLM Project, GGUF (vllm-gguf-plugin plugin, “highly experimental and under-optimized” support, base model tokenizer), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/quantization/gguf/ ↩

  6. vLLM Project, Parallelism and Scaling — Tensor Parallelism (--tensor-parallel-size, weight sharding, NVLink recommendations). https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ ↩ ↩2

  7. vLLM Project, Parallelism and Scaling — Multi-node deployment (run_cluster.sh, --distributed-executor-backend ray, --nnodes / --node-rank / --headless, InfiniBand and NCCL), accessed 2026-10-09. https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ ↩ ↩2 ↩3 ↩4

  8. vLLM Project, Disaggregated Prefill and Decode (--kv-transfer-config, NIXL / LMCache / Mooncake / Offloading connectors, experimental status), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/disagg_prefill/ ↩ ↩2 ↩3 ↩4

  9. vLLM Project, CLI Reference — vllm serve (exhaustive flag list: no --role, --request-timeout, --calculate-kv-cache-size; --speculative-config), accessed 2026-10-09 · vLLM Project, Release v0.29.0 (addition of --max-num-queued-reqs), 2026-09-09. https://docs.vllm.ai/en/stable/cli/serve/ · https://github.com/vllm-project/vllm/releases/tag/v0.29.0 ↩ ↩2 ↩3 ↩4

  10. vLLM Project, Release v0.32.0 (“Default CORS settings no longer allow every origin in the Python and Rust frontends; set --allowed-origins to permit browser origins”, #61009; empty list by default; fixes CVE-2026-115917, vLLM ≤ 0.31.x), 18 October 2026. https://github.com/vllm-project/vllm/releases/tag/v0.32.0 · https://github.com/vllm-project/vllm/pull/61009 ↩

  11. vLLM Project, Quantized KV Cache (--kv-cache-dtype fp8 / fp8_e5m2, CUDA 11.8+ and ROCm, llm-compressor calibration, --kv-cache-dtype-skip-layers), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache/ ↩

  12. vLLM Project, Automatic Prefix Caching (hashing by KV Cache blocks, TTFT impact, tensor parallelism compatibility), accessed 2026-10-10. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ ↩ ↩2 ↩3

  13. vLLM Project, Release v0.24.0 (vLLM no longer sets CUDA_VISIBLE_DEVICES; device_ids argument; variable deprecated on ROCm), 2026-06-29. https://github.com/vllm-project/vllm/releases/tag/v0.24.0 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.