Skip to content

Attention (mechanism)

πŸ“ Short definition

Computation that determines the relative importance of each token in context to build the representation of a given token β€” the heart of the Transformer.

πŸ“– Detailed definition

For each token, the model computes three vectors from its embedding:

  • Query (Q): β€œwhat is this token looking for?”
  • Key (K): β€œwhat does each other token contain?”
  • Value (V): β€œwhat information to pull from each other token?”

The attention score between two tokens is the dot product QΒ·K. Those scores weight the Values to produce the token’s final representation. This calculation repeats in every Transformer layer.

Link to KV Cache: during Prefill, K and V vectors for each token are computed and stored in the KV Cache. In Decoding, the model reuses them without recomputing.

Attention cost is quadratic O(nΒ²) in context length: doubling the prompt multiplies the cost of the attention part of prefill by 4 (the dense layers only cost 2 times more).

πŸ’‘ Why it matters for on-prem AI

The quadratic cost of attention at Prefill explains why very long prompts (100K+ tokens) spike TTFT; VRAM saturation comes instead from the KV Cache, which grows linearly with context β€” hence Flash Attention (computing without materializing the nΒ² matrix) and GQA (smaller KV Cache).

⚠️ Common pitfalls

  • O(nΒ²) applies to prefill. In decoding, the KV Cache brings cost to O(n) per generated token.
  • GQA (Grouped-Query Attention, used in Llama 3+) shrinks KV Cache size by sharing K/V heads across several Q heads β€” without removing attention itself.

πŸ“š Go deeper

  1. 🧠 The Journey of a Prompt (Step 3: prefill and attention in action)
  2. πŸ’Ύ KV Cache & context (what attention stores and why it weighs on VRAM)

πŸ”— See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.