π³ RadixAttention
π Short definition
KV Cache management technique that organizes key/value entries in a radix tree (prefix tree) rather than in a flat buffer β common prefixes across requests are computed only once.
π Detailed definition
Introduced by SGLang1, RadixAttention stores KV Cache entries in a tree structure: each node corresponds to a token segment. When multiple requests share an identical prefix β long system prompt, retrieved RAG context, repeated tool schema β the corresponding KV entries are stored once and reused by all requests that follow that path.
Primary effect: significant reduction in TTFT in agentic loops (the LLM calls tools multiple times with the same contextual framework) and in deployments where many users share the same system prompt.
π‘ Relation with PagedAttention
Both techniques optimize KV Cache memory management, but for different goals:
| Technique | Primary objective |
|---|---|
| PagedAttention | Eliminate memory fragmentation to increase concurrency |
| RadixAttention | Maximize prefix reuse to reduce latency |
They are not mutually exclusive conceptually: PagedAttention addresses throughput under load; RadixAttention addresses latency when context repeats.
π See also
Footnotes
-
LMSys, Fast and Expressive LLM Inference with RadixAttention, January 2024. https://lmsys.org/blog/2024-01-17-sglang/ β©