vLLM
๐ Short definition
Open-source inference engine (Python/C++) designed to serve LLMs at high throughput on dedicated GPUs, with advanced KV Cache management and concurrent requests1.
๐ Detailed definition
vLLM has become the on-premise reference for multi-user inference on servers with NVIDIA (CUDA) or AMD (ROCm) GPUs. Unlike workstation-oriented tools, vLLM targets production: OpenAI-compatible API, continuous batching, tensor parallelism, and FP8 (fp8_per_tensor shortcut since 0.31) / AWQ / GPTQ quantization for recent architectures2.
Its signature mechanism is the paged KV Cache management popularized by PagedAttention: the cache is split into reusable, hashed blocks (prefix caching enabled by default), reducing memory fragmentation and enabling many simultaneous requests without saturating VRAM34. The legacy implementation was removed in v0.25 (July 2026) in favor of the V1 engineโs attention backends, which keep this memory model5.
๐ก Why it matters for on-prem AI
- Scenario D in this vault (multi-GPU datacenter), and Scenario B in its NVIDIA GPU rack server variant (see GPU rack servers); the Mac / APU appliances of Scenario B (MLX, llama.cpp, SGLang ROCm) and the Mac cluster of Scenario C (MLX / Exo) do not rely on vLLM.
- Natural alternative to a cloud API when internal request volume justifies hardware amortization.
- Anchor point for configuring vLLM multi-GPU and migrating from Ollama.
โ ๏ธ Common pitfalls
- Deploying vLLM on a machine without a dedicated GPU or with massive RAM offloading: that is not its use case (prefer Ollama / llama.cpp).
- Comparing vLLM and Ollama on a single sequential request: vLLMโs advantage appears under concurrency.
- Forgetting VRAM sizing: model + concurrent KV Cache must fit in available GPU memory.
- Believing vLLM does not reuse prefixes: Automatic Prefix Caching is enabled by default4; on agentic workloads with highly shared prefixes, still evaluate SGLang in parallel (RadixAttention, finer-grained sharing).
- Exposing the server with
--api-keyalone: it does not cover all routes โ a reverse proxy is mandatory (see Local inference security). - Staying on a version earlier than 0.34.0 (16 November 2026): several remote code executions and denials of service were fixed between 0.28 and 0.33, followed by the chat template lock bypass via
/v1/systemonein 0.34; the wheels have targeted CUDA 13.0 by default since 0.28 (August 2026), and exclusively since 0.33, and 0.34 removes Model Runner V1 (VLLM_USE_V2_MODEL_RUNNER=0makes startup fail) and only publishes ROCm 10.0 builds: read the โBefore you upgradeโ section of the release notes before migrating2.
๐ To go deeper
- โ๏ธ Inference engines โ vLLM, Ollama, TensorRT-LLM, SGLang comparison
- PagedAttention โ vLLMโs key memory optimization
- ๐ Local inference security โ hardening API in production
๐ See also
Footnotes
-
vLLM Project, official repository. https://github.com/vllm-project/vllm โฉ
-
vLLM Project, Release v0.31.0 (per-request
mm_processor_kwargsrejected unless--trust-request-mm-kwargs; online quantization shortcutfp8renamedfp8_per_tensor), 5 October 2026 ยท Release v0.28.0 (default PyPI wheel and Docker image on CUDA 13.0), 26 August 2026 ยท Release v0.33.0 (CUDA 13.0 only wheels and images; fixes the advisories of 2 November 2026), 2 November 2026 ยท Release v0.34.0 (โBefore you upgradeโ: Model Runner V1 removed, ROCm 10.0 builds only), 16 November 2026 ยท advisory GHSA-xgh8-qwjh-q853 (/v1/systemonebypasses--trust-request-chat-template, Moderate, versions 0.32.0rc0 โ 0.33.0, fixed in 0.34.0), 18 November 2026. https://github.com/vllm-project/vllm/releases/tag/v0.31.0 ยท https://github.com/vllm-project/vllm/releases/tag/v0.28.0 ยท https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3 ยท https://github.com/vllm-project/vllm/releases/tag/v0.34.0 ยท https://github.com/vllm-project/vllm/security/advisories/GHSA-xgh8-qwjh-q853 โฉ โฉ2 -
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). https://arxiv.org/abs/2309.06180 โฉ
-
vLLM Project, Automatic Prefix Caching (
enable_prefix_cachingenabled by default, block hashing), accessed 2026-10-10. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ โฉ โฉ2 -
vLLM Project, Release v0.25.0 (โPagedAttention has been removed โ the legacy attention implementation is deleted now that V1/MRv2 backends are the standard pathโ), 11 July 2026. https://github.com/vllm-project/vllm/releases/tag/v0.25.0 โฉ