Linear attention
π Short definition
A variant of attention that keeps the memory of the context in a fixed-size state, updated at each token, instead of a KV cache that grows with the conversation.
π Detailed definition
With classic attention, each token adds its keys and values to the cache: memory grows in proportion to the context. Linear attention replaces this cache with a fixed-size notebook: at each token, the model erases a little, corrects and adds the essentials. The notebook does not grow, even over 1 million tokens.
Recent variants add a forgetting mechanism (gating) to erase quickly and a βdelta ruleβ to correct a specific piece of information (Gated DeltaNet)1. Because a fixed notebook retains fewer details than a full cache, models keep a few classic attention layers: this is called a hybrid architecture (Gated DeltaNet in Qwen3-Next and Qwen3.8-27B, Kimi Delta Attention in Kimi Linear, Lightning Attention in MiniMax-M1; ratios and gains in πΎ KV Cache & Context).
π‘ Why it matters for on-premise AI
These models accept very long contexts with far less cache VRAM. When sizing a server, the classic GQA formula overestimates them: only a minority of layers stores a real KV cache.
β οΈ Common pitfalls
- Applying the KV cache formula to every layer of the model: only the classic attention layers have one.
- Assuming every engine supports them: check your engineβs model list and its minimum version.
π To go deeper
- πΎ KV Cache & Context (section βWhen the model itself shrinks its memoryβ)
π See also
Footnotes
-
S. Yang, J. Kautz, A. Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule (arXiv:2412.06464), December 2024. https://arxiv.org/abs/2412.06464 β©