Skip to content

🌳 RadixAttention

πŸ“ Short definition

KV Cache management technique that organizes key/value entries in a radix tree (prefix tree) rather than in a flat buffer β€” common prefixes across requests are computed only once.

πŸ“– Detailed definition

Introduced by SGLang1, RadixAttention stores KV Cache entries in a tree structure: each node corresponds to a token segment. When multiple requests share an identical prefix β€” long system prompt, retrieved RAG context, repeated tool schema β€” the corresponding KV entries are stored once and reused by all requests that follow that path.

Primary effect: significant reduction in TTFT in agentic loops (the LLM calls tools multiple times with the same contextual framework) and in deployments where many users share the same system prompt.

πŸ’‘ Relation with PagedAttention

Both techniques optimize KV Cache memory management, but for different goals:

TechniquePrimary objective
PagedAttentionEliminate memory fragmentation to increase concurrency
RadixAttentionMaximize prefix reuse to reduce latency

They are not mutually exclusive conceptually: PagedAttention addresses throughput under load; RadixAttention addresses latency when context repeats.

πŸ”— See also

Footnotes

  1. LMSys, Fast and Expressive LLM Inference with RadixAttention, January 2024. https://lmsys.org/blog/2024-01-17-sglang/ ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.