Lexicon index
Auto-generated at build time. For a guided read, see AI Glossary.
| Term | Definition |
|---|---|
| โก SGLang | Open-source LLM inference and serving framework, an alternative to vLLM for agentic workloads and structured outputs. |
| ๐ณ RadixAttention | KV Cache management technique using a prefix tree, introduced by SGLang to reuse common contexts across requests. |
| ๐ข Multi-tenant | AI SaaS architecture where a single infrastructure serves multiple isolated organizations, with inter-tenant leakage risk in RAG. |
| ๐ Zero Data Retention (ZDR) | Cloud LLM API contractual clause: no persistence, reuse or human review of prompts and responses for the covered models and endpoints. |
| APU | Chip combining CPU, GPU, and NPU on one SoC with shared unified memory. |
| Attention (mechanism) | Core Transformer mechanism that lets each token weight the importance of other tokens in context. |
| Autonomous agent (LLM) | System where an LLM drives tools and decisions itself to complete a multi-step task. |
| Context window | Maximum number of tokens an LLM can handle in active input โ drives dynamic inference memory cost. |
| CUDA | NVIDIAโs software platform for programming its GPUs; most local AI engines target it first, which weighs on the choice of hardware for on-premise AI. |
| Custodian agent | Autonomous agent responsible for maintaining a vault, repository, or document corpus by proposing human-validated corrections. |
| Decoding | Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall. |
| ECN | Explicit Congestion Notification โ network congestion signaling used with RoCE to avoid packet loss. |
| Embedding | Dense numerical representation of a token or document in vector space. |
| Excessive Agency | OWASP Top 10 for LLM applications vulnerability: an AI agent has too many features, permissions or too much autonomy, allowing unintended real-world actions. |
| Exo | Open-source P2P orchestrator to merge memory across machines into a local AI cluster. |
| GGUF | Portable file format for local inference with llama.cpp, optimized for K-quant quantization. |
| GPUDirect RDMA | Mechanism letting GPUs exchange data directly with network devices without CPU copies. |
| GraphRAG | RAG evolution based on a knowledge graph rather than a vector database alone. |
| HBM | Stacked memory with very high bandwidth, used on professional AI accelerators. |
| Human-in-the-loop | Governance mode where an important automated action waits for human validation before it is applied. |
| InfiniBand | Dedicated high-performance network fabric for GPU clusters, HPC and AI datacenter standard. |
| KV Cache | Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length. |
| LangGraph | State-graph framework for orchestrating multi-step LLM agents with loops, memory, and flow control. |
| Linear attention | Family of attention that summarizes the context in a fixed-size state instead of a growing KV cache; often mixed with a few classic layers (hybrid). |
| LiteLLM | OpenAI-compatible gateway routing LLM calls to local or cloud models through one interface. |
| LLM | AI model trained on very large text corpora, most often of the Transformer type; locally, its performance depends as much on memory as on the model itself. |
| LLM benchmark | Standardized test suite to compare capabilities, limits, and risks of language models. |
| LLM inference | Phase where an already trained LLM produces a response token by token, in two steps (prefill then decoding); it is what sizes on-premise hardware. |
| LLM-as-a-judge | Evaluation technique where a language model judges or scores responses. |
| LPDDR5X | Low-power RAM standardized by JEDEC and soldered next to the chip; it powers unified-memory machines such as Macs, AMD Ryzen AI Max and DGX Spark. |
| Memory bandwidth | Amount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be. |
| Memory Tree | Memory architecture organizing documents and summaries in a hierarchy to limit context injected into the LLM. |
| Memory Wall | Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone. |
| MLA (Multi-head Latent Attention) | Attention variant that stores one small compressed vector per token in the KV cache instead of the full keys and values, introduced by DeepSeek-V2. |
| MoE | Mixture of Experts โ architecture activating only some sub-networks per token, enabling huge models with controlled inference cost. |
| Multi-GPU | Architecture using several GPUs in one machine or a cluster to increase memory and throughput; the gains depend on the interconnect (PCIe, NVLink). |
| NCCL | NVIDIA collective communication library optimized for large-scale GPU-to-GPU transfers. |
| NPU | Neural Processing Unit โ AI accelerator in modern SoCs, useful for some tasks but limited for large LLMs. |
| NVLink | Dedicated hardware link that connects NVIDIA GPUs at very high speed, bypassing the PCIe bus; it is gone from recent workstation and consumer cards. |
| NVSwitch | NVIDIA switch that connects multiple GPUs into a fully non-blocking NVLink fabric inside a node. |
| Offloading | Technique that places part of the model in RAM or on SSD when VRAM is insufficient, at the cost of reduced throughput. |
| Ollama | Simplified local runtime to download and run LLMs via llama.cpp or MLX, with OpenAI-compatible API. |
| On-Premise (AI) | AI infrastructure hosted and operated on the organizationโs own equipment, without delegating to a cloud provider. |
| PagedAttention | KV Cache management technique using virtual memory blocks, popularized by vLLM. |
| PCIe | Standard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink. |
| PFC | Priority Flow Control โ Ethernet per-priority pause mechanism for a lossless network required by RoCE. |
| Pipeline Parallelism | Strategy for distributing an LLM in layer slices across multiple machines. |
| Prefill | Inference phase that processes the initial prompt in parallel before word-by-word generation. |
| Prompt injection | Attack where untrusted content in the LLM context hijacks system instructions to exfiltrate data or execute unauthorized actions. |
| Q4 Quantization | The most widely used 4-bit quantization format for local inference โ especially Q4_K_M in the GGUF/Ollama ecosystem. |
| Quantization | Reducing numerical precision of LLM weights to lower memory footprint and speed up inference. |
| RAG | Architecture that combines document retrieval and LLM generation to ground answers in internal sources. |
| RAGAS | Framework that evaluates a RAG system by separating retrieval quality from answer faithfulness, to diagnose where the pipeline fails. |
| RAM | System main memory, second choice for LLM inference when VRAM is insufficient. |
| Ray | Distributed computing framework for multi-node LLM orchestration in production. |
| RDMA | Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations. |
| RoCE | Protocol bringing RDMA benefits to standard Ethernet, without a dedicated InfiniBand fabric, at the cost of configuring a lossless network (PFC, ECN). |
| SmolAgents | Lightweight Hugging Face framework for local agentic orchestrationโa sovereign alternative to LangChain. |
| SoC | System on a Chip: a single chip that combines processor, graphics, AI accelerator and memory controller; the basis of Apple Silicon Macs and DGX Spark. |
| Sparse attention | Attention where each token rereads only a selection of past tokens judged relevant, instead of the whole history, to cut the cost of long contexts. |
| Speculative Decoding | Inference acceleration technique where a small fast model proposes tokens that the large model verifies in a single pass; both models must be loaded. |
| Tensor Parallelism | Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node. |
| TensorRT-LLM | NVIDIA SDK for optimized LLM inference (native PyTorch, FP8/NVFP4) on datacenter GPUs. |
| TFLOPS | Unit measuring floating-point compute capacity per second; useful to compare raw compute, but not enough on its own to predict LLM performance. |
| Thunderbolt | High-throughput wired interface for workstations and desktop AI clusters. |
| Tokenization | Splitting text into numeric units (tokens) before an LLM processes it. |
| Tokens per second | Number of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware. |
| Tool calling (Function / Tool Calling) | An LLMโs ability to emit structured requests to external functions (API, SQL, code) rather than free-form text. |
| TTFT | Time To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric. |
| Unified Memory | Architecture where CPU, GPU and sometimes NPU share a single memory pool, avoiding some copies over PCIe; found on Apple Silicon and some recent AMD APUs. |
| Vector database | Database specialized in storing and searching embedding vectors for RAG. |
| vLLM | Open-source high-throughput LLM inference engine for NVIDIA/AMD GPUs, multi-user production standard. |
| VRAM | Very high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading. |