Skip to content

MLA (Multi-head Latent Attention)

πŸ“ Short definition

A way of computing attention that keeps a compressed summary of each token in memory (a β€œlatent vector”) instead of its full keys (K) and values (V). The KV cache becomes much smaller.

πŸ“– Detailed definition

In classic attention, each layer of the model stores, for each token, one K vector and one V vector per attention head. MLA compresses this information into a single small vector per token and per layer. At compute time, the model β€œunfolds” this vector to recover the K and V it needs.

An analogy: instead of keeping full notes on every speaker in a meeting, you keep a summary of a few lines from which you can recover the essentials.

MLA was introduced by DeepSeek-V2 (May 2024)1; for the quantified gains and the models that use it (DeepSeek-V3, Kimi K2), see πŸ’Ύ KV Cache & Context.

πŸ’‘ Why it matters for on-premise AI

On an on-premise server, the VRAM available for the cache limits conversation length and the number of concurrent users. A model with MLA uses far less cache per token than a classic model of comparable size. The usual GQA formula does not apply: it greatly overestimates the memory needed.

⚠️ Common pitfalls

  • Sizing an MLA model with the GQA formula (2Γ—LΓ—HkvΓ—D2 \times L \times H_{kv} \times D…): the result is heavily overestimated. Start from the model card or from the cache size reported by the engine.
  • Confusing MLA with sparse attention: MLA reduces the space taken by each token; sparse attention reduces the number of tokens reread.

πŸ“š To go deeper

πŸ”— See also

Footnotes

  1. DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv:2405.04434; KV cache compressed into a latent vector, βˆ’93.3% vs DeepSeek 67B, equivalent to GQA with 2.25 groups), May 2024. https://arxiv.org/abs/2405.04434 ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.