This glossary centralizes key on-prem AI concepts.
Use it as a quick entry point, then open detailed entries.
Full index: all lexicon entries (alphabetical list, updated at build).
πΆ Beginner β getting started
Not sure where to begin:
LLM β LLM inference β π§ The Journey of a Prompt β Prefill / Decoding β Memory Wall β ποΈ Memory Bandwidth
Recommended paths
- Inference performance: LLM inference, Memory Wall, Memory bandwidth, Prefill, Decoding, Tokens per second, TTFT.
- Memory & hardware: VRAM, RAM, Unified memory, PCIe, Multi-GPU, NVLink, NVSwitch.
- Production AI systems: Quantization, Q4 quantization, KV Cache, RAG, RDMA, RoCE, InfiniBand.
- Datacenter networking: PFC, ECN, NCCL, GPUDirect RDMA.
- Model evaluation: LLM benchmark, LLM-as-a-judge, RAGAS.
Acronym index
| Acronym | Meaning |
|---|
| APU | Accelerated Processing Unit |
| CUDA | Compute Unified Device Architecture (NVIDIA GPU platform) |
| ECN | Explicit Congestion Notification |
| HBM | High Bandwidth Memory |
| IB | InfiniBand |
| KV | Key-Value |
| LLM | Large Language Model |
| LPDDR5X | Low Power Double Data Rate 5X (low-power memory) |
| MLA | Multi-head Latent Attention |
| MoE | Mixture of Experts |
| NCCL | NVIDIA Collective Communications Library |
| NPU | Neural Processing Unit |
| PCIe | Peripheral Component Interconnect Express |
| PFC | Priority Flow Control |
| Q4 | 4-bit quantization |
| RAG | Retrieval-Augmented Generation |
| RAM | Random Access Memory |
| RDMA | Remote Direct Memory Access |
| RoCE | RDMA over Converged Ethernet |
| SoC | System on a Chip |
| TFLOPS | Tera Floating Point Operations Per Second |
| TTFT | Time To First Token |
| VRAM | Video RAM |
LLM foundations
| Term | Quick definition |
|---|
| Tokenisation | Splitting text into numeric units the model processes. |
| Embedding | Coordinate vector encoding a tokenβs meaning. |
| Attention | Mechanism weighting each tokenβs importance in context. |
| Term | Quick definition |
|---|
| Memory Wall | Performance limit from memory transfers rather than compute. |
| Memory bandwidth | Memory data transfer rate, often in GB/s. |
| Prefill | Prompt ingestion phase, more parallel. |
| Decoding | Autoregressive token-by-token phase, often memory-bound. |
| KV Cache | Cached attention keys/values to speed generation. |
| MLA | Attention that stores a compressed summary per token instead of full keys/values. |
| Sparse attention | Each token rereads only a selection of relevant past tokens. |
| Linear attention | Context memory kept in a fixed-size state instead of a growing cache. |
| Tokens/s | Generation throughput in tokens per second. |
Software stack
| Term | Quick definition |
|---|
| Ollama | Local llama.cpp runtime + OpenAI API for testing and lab use. |
| vLLM | High-throughput production engine, PagedAttention, multi-user. |
| SGLang | Agentic inference, RadixAttention, structured JSON. |
| RadixAttention | Prefix-tree KV Cache (SGLang). |
| Speculative Decoding | Small draft model + large validator model to speed decoding. |
| PagedAttention | KV Cache managed in virtual blocks; key to vLLM production. |
| GGUF | Portable format for local inference with llama.cpp/Ollama. |
| TensorRT-LLM | NVIDIA SDK for highly optimized datacenter GPU compilation. |
| Exo | P2P orchestrator for desktop AI clusters (Mac Mini + Thunderbolt). |
| Ray | Distributed framework for multi-node production deployment. |
| Pipeline Parallelism | Splitting a model into layer slices across nodes. |
| Tensor Parallelism | Splitting matrices within one NVLink multi-GPU node. |
| Thunderbolt | Desktop interconnect up to 80 Gb/s for Exo clusters. |
Infrastructure & architecture
| Term | Quick definition |
|---|
| Unified memory | Shared CPU/GPU/NPU memory pool in an SoC. |
| Offloading | Partial move of weights/activations across memory tiers by capacity. |
| Multi-GPU | Several GPUs for more capacity and/or throughput. |
| NVLink | High-bandwidth GPU interconnect (pro/datacenter tiers). |
| NVSwitch | Non-blocking NVLink fabric among all GPUs in an HGX node. |
| PCIe | Standard CPUβGPU bus; inter-GPU bottleneck without NVLink. |
Networking & clustering
| Term | Quick definition |
|---|
| RDMA | Memory-to-memory transfer without CPU copy for multi-node clusters. |
| RoCE | RDMA over Ethernet; needs PFC + ECN for lossless operation. |
| InfiniBand | Dedicated HPC/AI network fabric, natively lossless, ~400β800 Gb/s. |
| GPUDirect RDMA | Direct VRAMβNIC transfer without CPU copy. |
| NCCL | NVIDIA library for GPU-to-GPU collective communication. |
| PFC | Priority Flow Control β per-priority pause for lossless RoCE. |
| ECN | Explicit Congestion Notification β proactive congestion handling. |
Foundational concepts
| Term | Quick definition |
|---|
| On-Premise | AI infrastructure hosted on the organizationβs own hardware. |
| LLM | Large language model trained on vast text corpora. |
| LLM inference | Running a trained LLM to generate text on demand. |
Applied AI
| Term | Quick definition |
|---|
| LLM inference | Running a trained LLM to generate text on demand. |
| RAG | Architecture enriching LLM context from a document base. |
| Vector database | Store and search documents by embedding similarity. |
| GraphRAG | RAG over a knowledge graph for global queries. |
| Memory Tree | Hierarchical memory loading only useful summaries and nodes. |
| Autonomous agent | LLM with tools that loops to complete tasks. |
| Custodian agent | Maintenance agent proposing corrections awaiting human validation. |
| Human-in-the-loop | Governance where humans validate critical actions before apply. |
| Excessive Agency | OWASP LLM06: agent too permissive in tools, rights, or autonomy. |
| LiteLLM | OpenAI-compatible gateway routing to local or cloud models. |
| SmolAgents | Lightweight Hugging Face agent framework without cloud telemetry. |
| LangGraph | State graph for multi-step agents and agentic RAG. |
| Tool calling | Function / tool calling β LLM to structured external functions. |
| Prompt injection | OWASP LLM01 β hijacking via untrusted content (RAG, web). |
| Zero Data Retention | ZDR clause: no cloud persistence of prompts. |
| Multi-tenant | Multi-organization isolation; inter-tenant RAG risk. |
| Context window | Maximum tokens the model can handle in active input. |
| Quantization | Lower numeric precision to cut memory and compute cost. |
| MoE | Architecture with specialized experts partially activated per token. |
Model evaluation
| Term | Quick definition |
|---|
| LLM benchmark | Standardized tests to compare language models. |
| LLM-as-a-judge | Using an LLM as judge to score or compare answers. |
| RAGAS | Framework evaluating RAG pipelines: retrieval, faithfulness, relevance. |
AI model trained on very large text corpora, most often of the Transformer type; locally, its performance depends as much on memory as on the model itself.Phase where an already trained LLM produces a response token by token, in two steps (prefill then decoding); it is what sizes on-premise hardware.Inference phase that processes the initial prompt in parallel before word-by-word generation.Generation phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.Situation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.Amount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be.Number of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware.Time To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric.Very high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading.System main memory, second choice for LLM inference when VRAM is insufficient.Architecture where CPU, GPU and sometimes NPU share a single memory pool, avoiding some copies over PCIe; found on Apple Silicon and some recent AMD APUs.Standard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink.Architecture using several GPUs in one machine or a cluster to increase memory and throughput; the gains depend on the interconnect (PCIe, NVLink).Dedicated hardware link that connects NVIDIA GPUs at very high speed, bypassing the PCIe bus; it is gone from recent workstation and consumer cards.NVIDIA switch that connects multiple GPUs into a fully non-blocking NVLink fabric inside a node.Reducing numerical precision of LLM weights to lower memory footprint and speed up inference.The most widely used 4-bit quantization format for local inference β especially Q4_K_M in the GGUF/Ollama ecosystem.Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.Architecture that combines document retrieval and LLM generation to ground answers in internal sources.Network technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.Protocol bringing RDMA benefits to standard Ethernet, without a dedicated InfiniBand fabric, at the cost of configuring a lossless network (PFC, ECN).Dedicated high-performance network fabric for GPU clusters, HPC and AI datacenter standard.Priority Flow Control β Ethernet per-priority pause mechanism for a lossless network required by RoCE.Explicit Congestion Notification β network congestion signaling used with RoCE to avoid packet loss.NVIDIA collective communication library optimized for large-scale GPU-to-GPU transfers.Mechanism letting GPUs exchange data directly with network devices without CPU copies.Standardized test suite to compare capabilities, limits, and risks of language models.Evaluation technique where a language model judges or scores responses.Framework that evaluates a RAG system by separating retrieval quality from answer faithfulness, to diagnose where the pipeline fails.Chip combining CPU, GPU, and NPU on one SoC with shared unified memory.NVIDIA's software platform for programming its GPUs; most local AI engines target it first, which weighs on the choice of hardware for on-premise AI.Stacked memory with very high bandwidth, used on professional AI accelerators.Low-power RAM standardized by JEDEC and soldered next to the chip; it powers unified-memory machines such as Macs, AMD Ryzen AI Max and DGX Spark.Attention variant that stores one small compressed vector per token in the KV cache instead of the full keys and values, introduced by DeepSeek-V2.Mixture of Experts β architecture activating only some sub-networks per token, enabling huge models with controlled inference cost.Neural Processing Unit β AI accelerator in modern SoCs, useful for some tasks but limited for large LLMs.System on a Chip: a single chip that combines processor, graphics, AI accelerator and memory controller; the basis of Apple Silicon Macs and DGX Spark.Unit measuring floating-point compute capacity per second; useful to compare raw compute, but not enough on its own to predict LLM performance.Splitting text into numeric units (tokens) before an LLM processes it.Dense numerical representation of a token or document in vector space.Core Transformer mechanism that lets each token weight the importance of other tokens in context.Attention where each token rereads only a selection of past tokens judged relevant, instead of the whole history, to cut the cost of long contexts.Family of attention that summarizes the context in a fixed-size state instead of a growing KV cache; often mixed with a few classic layers (hybrid).Simplified local runtime to download and run LLMs via llama.cpp or MLX, with OpenAI-compatible API.Open-source high-throughput LLM inference engine for NVIDIA/AMD GPUs, multi-user production standard.Open-source LLM inference and serving framework, an alternative to vLLM for agentic workloads and structured outputs.KV Cache management technique using a prefix tree, introduced by SGLang to reuse common contexts across requests.Inference acceleration technique where a small fast model proposes tokens that the large model verifies in a single pass; both models must be loaded.KV Cache management technique using virtual memory blocks, popularized by vLLM.Portable file format for local inference with llama.cpp, optimized for K-quant quantization.NVIDIA SDK for optimized LLM inference (native PyTorch, FP8/NVFP4) on datacenter GPUs.Open-source P2P orchestrator to merge memory across machines into a local AI cluster.Distributed computing framework for multi-node LLM orchestration in production.Strategy for distributing an LLM in layer slices across multiple machines.Strategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.High-throughput wired interface for workstations and desktop AI clusters.Technique that places part of the model in RAM or on SSD when VRAM is insufficient, at the cost of reduced throughput.AI infrastructure hosted and operated on the organization's own equipment, without delegating to a cloud provider.Database specialized in storing and searching embedding vectors for RAG.RAG evolution based on a knowledge graph rather than a vector database alone.Memory architecture organizing documents and summaries in a hierarchy to limit context injected into the LLM.System where an LLM drives tools and decisions itself to complete a multi-step task.Autonomous agent responsible for maintaining a vault, repository, or document corpus by proposing human-validated corrections.Governance mode where an important automated action waits for human validation before it is applied.OWASP Top 10 for LLM applications vulnerability: an AI agent has too many features, permissions or too much autonomy, allowing unintended real-world actions.OpenAI-compatible gateway routing LLM calls to local or cloud models through one interface.Lightweight Hugging Face framework for local agentic orchestrationβa sovereign alternative to LangChain.State-graph framework for orchestrating multi-step LLM agents with loops, memory, and flow control.An LLM's ability to emit structured requests to external functions (API, SQL, code) rather than free-form text.Attack where untrusted content in the LLM context hijacks system instructions to exfiltrate data or execute unauthorized actions.Cloud LLM API contractual clause: no persistence, reuse or human review of prompts and responses for the covered models and endpoints.AI SaaS architecture where a single infrastructure serves multiple isolated organizations, with inter-tenant leakage risk in RAG.Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.