Skip to content

KV Cache

πŸ“ Short definition

Memory holding already-computed attention states to avoid costly recomputation.

πŸ“– Detailed definition

In a Transformer, each new token reuses keys/values from prior tokens. The KV cache speeds that reuse but increases memory use as context grows.

πŸ’‘ Why it matters for on-prem AI

It strongly affects practical context capacity and stability on long conversations.

πŸ”¬ How it works (not magic)

Each generated token must consult all past context. Instead of recomputing history from scratch (quadratic cost), the model stores intermediate states (attention layer keys and values) computed during prefill β€” in VRAM.

πŸ“š Go deeper

Want to know why this cache can saturate your machine?

  1. 🧠 The Journey of a Prompt (Step 4: how KV cache fills during prefill)
  2. πŸ’Ύ KV Cache & context (full chapter: formula, GQA, PagedAttention)
  3. 🏎️ Memory bandwidth (why cache size slows generation)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.