Skip to content

Sparse attention

πŸ“ Short definition

A variant of attention where each new token looks at only part of the previous tokens, chosen for their relevance, instead of rereading the whole context.

πŸ“– Detailed definition

In classic attention, each token compares its content with all previous tokens. The cost grows with the square of the context length. Sparse attention limits this work to a selection.

An analogy: to find a piece of information in a long file, you read the table of contents, then open only the useful pages.

Example: DeepSeek Sparse Attention (DSA) in DeepSeek-V3.21, detailed in πŸ’Ύ KV Cache & Context.

πŸ’‘ Why it matters for on-premise AI

Sparse attention makes long contexts (100K tokens and more) cheaper to compute and faster. On an on-premise server, this means more long requests handled with the same hardware.

⚠️ Common pitfalls

  • Believing that sparse attention reduces cache VRAM: with DSA, all tokens remain stored in the KV cache (in the compressed form of MLA)1. What goes down is the amount of cache reread and computed for each token.

πŸ“š To go deeper

πŸ”— See also

Footnotes

  1. DeepSeek-AI, DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv:2512.02556, section 2.1: lightning indexer, 2,048 selected tokens, O(LΒ²) β†’ O(Lk), DSA implemented on MLA), December 2025. https://arxiv.org/abs/2512.02556 ↩ ↩2

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.