Memory Tree
π Short definition
Long-memory pattern where documents are chunked, summarized, and arranged in a hierarchy so the agent loads only useful nodes into context.
π Detailed definition
A Memory Tree replaces or complements a classic vector database. Instead of only retrieving chunks similar to a question, the agent walks a hierarchy: titles, short summaries, sub-summaries, then exact content if needed.
The goal is to reduce injected tokens and thus pressure on the context window and KV Cache.
π‘ Why it matters for on-prem AI
On a local machine, VRAM and memory bandwidth are limited. A Memory Tree helps keep context short, lowering TTFT and avoiding saturating the server with full documents.
β οΈ Common pitfalls
- Assuming a Memory Tree replaces every vector-database use case.
- Trusting stale summaries if the tree is not regenerated.
- Forgetting to cite the exact source document after tree navigation.
π Go deeper
- RAG & Agents: Memory Tree approach
- OpenHuman (popularized the pattern in its early versions, no longer implements it)
π See also
Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.Time To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric.Architecture that combines document retrieval and LLM generation to ground answers in internal sources.