Skip to content

PagedAttention

πŸ“ Short definition

Algorithm that manages KV Cache memory in non-contiguous blocks (like OS virtual memory), greatly reducing fragmentation.

πŸ“– Detailed definition

In classic engines, the KV Cache is pre-allocated in contiguous VRAM blocks: unused regions stay wasted. PagedAttention splits the cache into fixed-size pages that can be allocated, freed, and shared dynamically.

Result: memory waste drops from 60–80% to under 4% according to the original paper (Woosuk Kwon et al., SOSP 2023)1. This enables Continuous Batching: requests are processed continuously without draining the server between each one, maximizing GPU throughput.

πŸ’‘ Why it matters for on-prem AI

This is the innovation that made vLLM’s continuous batching possible β€” together with block-based prefix caching (enabled by default), it explains why vLLM outperforms Ollama in multi-user production2. Without PagedAttention, the server wastes VRAM and cannot batch concurrent requests efficiently.

⚠️ Common pitfalls

  • PagedAttention now designates a principle more than a component: vLLM removed the historical implementation in v0.25 (July 2026) in favor of its V1 engine’s attention backends, which keep block-based KV Cache management (--block-size option, vllm:kv_cache_usage_perc metric)3. Workstation-oriented engines rely on a different memory model: llama-server (llama.cpp) does do continuous batching (enabled by default, parallel slots via -np), but without a paged KV Cache or block sharing between requests, and Ollama by default processes only one request at a time per model (OLLAMA_NUM_PARALLEL=1).
  • Does not remove total capacity limits: if model + caches exceed total VRAM, OOM still occurs.

πŸ“š Go deeper

  1. βš™οΈ Inference Engines (why vLLM > Ollama in production)
  2. πŸ’Ύ KV Cache & Context (the cache mechanism PagedAttention optimizes)

πŸ”— See also

Footnotes

  1. W. Kwon et al., vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (β€œexisting systems waste 60% – 80% of memory”, β€œa mere waste of under 4%”), 20 June 2023; Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). https://vllm.ai/blog/2023-06-20-vllm Β· https://arxiv.org/abs/2309.06180 ↩

  2. vLLM Project, Automatic Prefix Caching (enable_prefix_caching enabled by default, block hashing), accessed 2026-10-10. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ ↩

  3. vLLM Project, Release v0.25.0 (β€œPagedAttention has been removed β€” the legacy attention implementation is deleted now that V1/MRv2 backends are the standard path”), 11 July 2026. https://github.com/vllm-project/vllm/releases/tag/v0.25.0 Β· vLLM Project, Engine Arguments (--block-size, block-based KV cache), accessed 2026-10-10. https://docs.vllm.ai/en/stable/configuration/engine_args/ ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.