Skip to content

Context window

📝 Short definition

Maximum number of tokens (prompt + history + response in progress) the model can “see” and use during inference. Beyond that limit, oldest tokens are ignored or truncated.

📖 Detailed definition

The context window determines two distinct things:

  1. What the model can read: if your document is 50,000 tokens and the model has a 32K window, it cannot read everything in one request.
  2. Dynamic VRAM consumed: each token in context uses space in the KV Cache. Longer windows mean a larger KV Cache.

Rough KV Cache sizes for a 70B model (Llama 3.1: 80 layers, 8 KV heads in GQA, dimension 128), for a single request, by context length:

ContextEstimated KV Cache (BF16)KV Cache (Q8)
4K tokens~1.3 GB~0.7 GB
32K tokens~10.7 GB~5.4 GB
128K tokens~42 GB~21 GB

A 128K context can double VRAM needs versus weights quantized to Q4 (~40 GB of weights + ~42 GB of KV Cache in BF16). In practice, context window size causes unexpected OOM on local machines.

💡 Why it matters for on-prem AI

On a 128 GB APU (~120 GB allocatable to the GPU on Mac via sysctl), a 70B Q4 model (~40 GB) leaves ~80 GB for KV Cache: the native 128K-token window (~42 GB in BF16) fits for one request, with ~38 GB of headroom. Comfortable for most uses. On a 24 GB VRAM machine, KV Cache is the first knob to adjust.

⚠️ Common pitfalls

  • Buying a “128K context” model assuming 4K and 128K perform the same: throughput (TTFT especially) degrades sharply on long contexts.
  • Raising context via --ctx-size without checking available VRAM: OOM hits mid-conversation, not at launch.
  • Treating RAG and context window as equivalent: RAG picks relevant passages to stay in a short window; a long window reads everything at once but costs more.

📚 Go deeper

🔗 See also

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue ↗

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.