Sparse attention
π Short definition
A variant of attention where each new token looks at only part of the previous tokens, chosen for their relevance, instead of rereading the whole context.
π Detailed definition
In classic attention, each token compares its content with all previous tokens. The cost grows with the square of the context length. Sparse attention limits this work to a selection.
An analogy: to find a piece of information in a long file, you read the table of contents, then open only the useful pages.
Example: DeepSeek Sparse Attention (DSA) in DeepSeek-V3.21, detailed in πΎ KV Cache & Context.
π‘ Why it matters for on-premise AI
Sparse attention makes long contexts (100K tokens and more) cheaper to compute and faster. On an on-premise server, this means more long requests handled with the same hardware.
β οΈ Common pitfalls
- Believing that sparse attention reduces cache VRAM: with DSA, all tokens remain stored in the KV cache (in the compressed form of MLA)1. What goes down is the amount of cache reread and computed for each token.
π To go deeper
- πΎ KV Cache & Context (section βWhen the model itself shrinks its memoryβ)
π See also
Footnotes
-
DeepSeek-AI, DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv:2512.02556, section 2.1: lightning indexer, 2,048 selected tokens, O(LΒ²) β O(Lk), DSA implemented on MLA), December 2025. https://arxiv.org/abs/2512.02556 β© β©2