KV Cache
π Short definition
Memory holding already-computed attention states to avoid costly recomputation.
π Detailed definition
In a Transformer, each new token reuses keys/values from prior tokens. The KV cache speeds that reuse but increases memory use as context grows.
π‘ Why it matters for on-prem AI
It strongly affects practical context capacity and stability on long conversations.
π¬ How it works (not magic)
Each generated token must consult all past context. Instead of recomputing history from scratch (quadratic cost), the model stores intermediate states (attention layer keys and values) computed during prefill β in VRAM.
π Go deeper
Want to know why this cache can saturate your machine?
- π§ The Journey of a Prompt (Step 4: how KV cache fills during prefill)
- πΎ KV Cache & context (full chapter: formula, GQA, PagedAttention)
- ποΈ Memory bandwidth (why cache size slows generation)
π See also
Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.