Skip to content

vLLM

๐Ÿ“ Short definition

Open-source inference engine (Python/C++) designed to serve LLMs at high throughput on dedicated GPUs, with advanced KV Cache management and concurrent requests1.

๐Ÿ“– Detailed definition

vLLM has become the on-premise reference for multi-user inference on servers with NVIDIA (CUDA) or AMD (ROCm) GPUs. Unlike workstation-oriented tools, vLLM targets production: OpenAI-compatible API, continuous batching, tensor parallelism, and FP8 (fp8_per_tensor shortcut since 0.31) / AWQ / GPTQ quantization for recent architectures2.

Its signature mechanism is the paged KV Cache management popularized by PagedAttention: the cache is split into reusable, hashed blocks (prefix caching enabled by default), reducing memory fragmentation and enabling many simultaneous requests without saturating VRAM34. The legacy implementation was removed in v0.25 (July 2026) in favor of the V1 engineโ€™s attention backends, which keep this memory model5.

๐Ÿ’ก Why it matters for on-prem AI

  • Scenario D in this vault (multi-GPU datacenter), and Scenario B in its NVIDIA GPU rack server variant (see GPU rack servers); the Mac / APU appliances of Scenario B (MLX, llama.cpp, SGLang ROCm) and the Mac cluster of Scenario C (MLX / Exo) do not rely on vLLM.
  • Natural alternative to a cloud API when internal request volume justifies hardware amortization.
  • Anchor point for configuring vLLM multi-GPU and migrating from Ollama.

โš ๏ธ Common pitfalls

  • Deploying vLLM on a machine without a dedicated GPU or with massive RAM offloading: that is not its use case (prefer Ollama / llama.cpp).
  • Comparing vLLM and Ollama on a single sequential request: vLLMโ€™s advantage appears under concurrency.
  • Forgetting VRAM sizing: model + concurrent KV Cache must fit in available GPU memory.
  • Believing vLLM does not reuse prefixes: Automatic Prefix Caching is enabled by default4; on agentic workloads with highly shared prefixes, still evaluate SGLang in parallel (RadixAttention, finer-grained sharing).
  • Exposing the server with --api-key alone: it does not cover all routes โ€” a reverse proxy is mandatory (see Local inference security).
  • Staying on a version earlier than 0.34.0 (16 November 2026): several remote code executions and denials of service were fixed between 0.28 and 0.33, followed by the chat template lock bypass via /v1/systemone in 0.34; the wheels have targeted CUDA 13.0 by default since 0.28 (August 2026), and exclusively since 0.33, and 0.34 removes Model Runner V1 (VLLM_USE_V2_MODEL_RUNNER=0 makes startup fail) and only publishes ROCm 10.0 builds: read the โ€œBefore you upgradeโ€ section of the release notes before migrating2.

๐Ÿ“š To go deeper

  1. โš™๏ธ Inference engines โ€” vLLM, Ollama, TensorRT-LLM, SGLang comparison
  2. PagedAttention โ€” vLLMโ€™s key memory optimization
  3. ๐Ÿ” Local inference security โ€” hardening API in production

๐Ÿ”— See also

Footnotes

  1. vLLM Project, official repository. https://github.com/vllm-project/vllm โ†ฉ

  2. vLLM Project, Release v0.31.0 (per-request mm_processor_kwargs rejected unless --trust-request-mm-kwargs; online quantization shortcut fp8 renamed fp8_per_tensor), 5 October 2026 ยท Release v0.28.0 (default PyPI wheel and Docker image on CUDA 13.0), 26 August 2026 ยท Release v0.33.0 (CUDA 13.0 only wheels and images; fixes the advisories of 2 November 2026), 2 November 2026 ยท Release v0.34.0 (โ€œBefore you upgradeโ€: Model Runner V1 removed, ROCm 10.0 builds only), 16 November 2026 ยท advisory GHSA-xgh8-qwjh-q853 (/v1/systemone bypasses --trust-request-chat-template, Moderate, versions 0.32.0rc0 โ†’ 0.33.0, fixed in 0.34.0), 18 November 2026. https://github.com/vllm-project/vllm/releases/tag/v0.31.0 ยท https://github.com/vllm-project/vllm/releases/tag/v0.28.0 ยท https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3 ยท https://github.com/vllm-project/vllm/releases/tag/v0.34.0 ยท https://github.com/vllm-project/vllm/security/advisories/GHSA-xgh8-qwjh-q853 โ†ฉ โ†ฉ2

  3. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). https://arxiv.org/abs/2309.06180 โ†ฉ

  4. vLLM Project, Automatic Prefix Caching (enable_prefix_caching enabled by default, block hashing), accessed 2026-10-10. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ โ†ฉ โ†ฉ2

  5. vLLM Project, Release v0.25.0 (โ€œPagedAttention has been removed โ€” the legacy attention implementation is deleted now that V1/MRv2 backends are the standard pathโ€), 11 July 2026. https://github.com/vllm-project/vllm/releases/tag/v0.25.0 โ†ฉ

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue โ†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.