Context window
📝 Short definition
Maximum number of tokens (prompt + history + response in progress) the model can “see” and use during inference. Beyond that limit, oldest tokens are ignored or truncated.
📖 Detailed definition
The context window determines two distinct things:
- What the model can read: if your document is 50,000 tokens and the model has a 32K window, it cannot read everything in one request.
- Dynamic VRAM consumed: each token in context uses space in the KV Cache. Longer windows mean a larger KV Cache.
Rough KV Cache sizes for a 70B model (Llama 3.1: 80 layers, 8 KV heads in GQA, dimension 128), for a single request, by context length:
| Context | Estimated KV Cache (BF16) | KV Cache (Q8) |
|---|---|---|
| 4K tokens | ~1.3 GB | ~0.7 GB |
| 32K tokens | ~10.7 GB | ~5.4 GB |
| 128K tokens | ~42 GB | ~21 GB |
A 128K context can double VRAM needs versus weights quantized to Q4 (~40 GB of weights + ~42 GB of KV Cache in BF16). In practice, context window size causes unexpected OOM on local machines.
💡 Why it matters for on-prem AI
On a 128 GB APU (~120 GB allocatable to the GPU on Mac via sysctl), a 70B Q4 model (~40 GB) leaves ~80 GB for KV Cache: the native 128K-token window (~42 GB in BF16) fits for one request, with ~38 GB of headroom. Comfortable for most uses. On a 24 GB VRAM machine, KV Cache is the first knob to adjust.
⚠️ Common pitfalls
- Buying a “128K context” model assuming 4K and 128K perform the same: throughput (TTFT especially) degrades sharply on long contexts.
- Raising context via
--ctx-sizewithout checking available VRAM: OOM hits mid-conversation, not at launch. - Treating RAG and context window as equivalent: RAG picks relevant passages to stay in a short window; a long window reads everything at once but costs more.