Skip to content

Linear attention

πŸ“ Short definition

A variant of attention that keeps the memory of the context in a fixed-size state, updated at each token, instead of a KV cache that grows with the conversation.

πŸ“– Detailed definition

With classic attention, each token adds its keys and values to the cache: memory grows in proportion to the context. Linear attention replaces this cache with a fixed-size notebook: at each token, the model erases a little, corrects and adds the essentials. The notebook does not grow, even over 1 million tokens.

Recent variants add a forgetting mechanism (gating) to erase quickly and a β€œdelta rule” to correct a specific piece of information (Gated DeltaNet)1. Because a fixed notebook retains fewer details than a full cache, models keep a few classic attention layers: this is called a hybrid architecture (Gated DeltaNet in Qwen3-Next and Qwen3.8-27B, Kimi Delta Attention in Kimi Linear, Lightning Attention in MiniMax-M1; ratios and gains in πŸ’Ύ KV Cache & Context).

πŸ’‘ Why it matters for on-premise AI

These models accept very long contexts with far less cache VRAM. When sizing a server, the classic GQA formula overestimates them: only a minority of layers stores a real KV cache.

⚠️ Common pitfalls

  • Applying the KV cache formula to every layer of the model: only the classic attention layers have one.
  • Assuming every engine supports them: check your engine’s model list and its minimum version.

πŸ“š To go deeper

πŸ”— See also

Footnotes

  1. S. Yang, J. Kautz, A. Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule (arXiv:2412.06464), December 2024. https://arxiv.org/abs/2412.06464 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.