Attention (mechanism)
π Short definition
Computation that determines the relative importance of each token in context to build the representation of a given token β the heart of the Transformer.
π Detailed definition
For each token, the model computes three vectors from its embedding:
- Query (Q): βwhat is this token looking for?β
- Key (K): βwhat does each other token contain?β
- Value (V): βwhat information to pull from each other token?β
The attention score between two tokens is the dot product QΒ·K. Those scores weight the Values to produce the tokenβs final representation. This calculation repeats in every Transformer layer.
Link to KV Cache: during Prefill, K and V vectors for each token are computed and stored in the KV Cache. In Decoding, the model reuses them without recomputing.
Attention cost is quadratic O(nΒ²) in context length: doubling the prompt multiplies the cost of the attention part of prefill by 4 (the dense layers only cost 2 times more).
π‘ Why it matters for on-prem AI
The quadratic cost of attention at Prefill explains why very long prompts (100K+ tokens) spike TTFT; VRAM saturation comes instead from the KV Cache, which grows linearly with context β hence Flash Attention (computing without materializing the nΒ² matrix) and GQA (smaller KV Cache).
β οΈ Common pitfalls
- O(nΒ²) applies to prefill. In decoding, the KV Cache brings cost to O(n) per generated token.
- GQA (Grouped-Query Attention, used in Llama 3+) shrinks KV Cache size by sharing K/V heads across several Q heads β without removing attention itself.
π Go deeper
- π§ The Journey of a Prompt (Step 3: prefill and attention in action)
- πΎ KV Cache & context (what attention stores and why it weighs on VRAM)