Skip to content

Lexicon index

Auto-generated at build time. For a guided read, see AI Glossary.

TermDefinition
โšก SGLangOpen-source LLM inference and serving framework, an alternative to vLLM for agentic workloads and structured outputs.
๐ŸŒณ RadixAttentionKV Cache management technique using a prefix tree, introduced by SGLang to reuse common contexts across requests.
๐Ÿข Multi-tenantAI SaaS architecture where a single infrastructure serves multiple isolated organizations, with inter-tenant leakage risk in RAG.
๐Ÿ” Zero Data Retention (ZDR)Cloud LLM API contractual clause: no persistence, reuse or human review of prompts and responses for the covered models and endpoints.
APUChip combining CPU, GPU, and NPU on one SoC with shared unified memory.
Attention (mechanism)Core Transformer mechanism that lets each token weight the importance of other tokens in context.
Autonomous agent (LLM)System where an LLM drives tools and decisions itself to complete a multi-step task.
Context windowMaximum number of tokens an LLM can handle in active input โ€” drives dynamic inference memory cost.
CUDANVIDIAโ€™s software platform for programming its GPUs; most local AI engines target it first, which weighs on the choice of hardware for on-premise AI.
Custodian agentAutonomous agent responsible for maintaining a vault, repository, or document corpus by proposing human-validated corrections.
DecodingGeneration phase where the model predicts one token at a time by rereading the KV Cache; it governs tokens per second and runs into the Memory Wall.
ECNExplicit Congestion Notification โ€” network congestion signaling used with RoCE to avoid packet loss.
EmbeddingDense numerical representation of a token or document in vector space.
Excessive AgencyOWASP Top 10 for LLM applications vulnerability: an AI agent has too many features, permissions or too much autonomy, allowing unintended real-world actions.
ExoOpen-source P2P orchestrator to merge memory across machines into a local AI cluster.
GGUFPortable file format for local inference with llama.cpp, optimized for K-quant quantization.
GPUDirect RDMAMechanism letting GPUs exchange data directly with network devices without CPU copies.
GraphRAGRAG evolution based on a knowledge graph rather than a vector database alone.
HBMStacked memory with very high bandwidth, used on professional AI accelerators.
Human-in-the-loopGovernance mode where an important automated action waits for human validation before it is applied.
InfiniBandDedicated high-performance network fabric for GPU clusters, HPC and AI datacenter standard.
KV CacheMemory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.
LangGraphState-graph framework for orchestrating multi-step LLM agents with loops, memory, and flow control.
Linear attentionFamily of attention that summarizes the context in a fixed-size state instead of a growing KV cache; often mixed with a few classic layers (hybrid).
LiteLLMOpenAI-compatible gateway routing LLM calls to local or cloud models through one interface.
LLMAI model trained on very large text corpora, most often of the Transformer type; locally, its performance depends as much on memory as on the model itself.
LLM benchmarkStandardized test suite to compare capabilities, limits, and risks of language models.
LLM inferencePhase where an already trained LLM produces a response token by token, in two steps (prefill then decoding); it is what sizes on-premise hardware.
LLM-as-a-judgeEvaluation technique where a language model judges or scores responses.
LPDDR5XLow-power RAM standardized by JEDEC and soldered next to the chip; it powers unified-memory machines such as Macs, AMD Ryzen AI Max and DGX Spark.
Memory bandwidthAmount of data transferred per second between memory and the processor or GPU, in GB/s; the key indicator for estimating how smooth text generation will be.
Memory TreeMemory architecture organizing documents and summaries in a hierarchy to limit context injected into the LLM.
Memory WallSituation where memory throughput limits performance more than raw compute power, especially in autoregressive generation; not to be judged on TFLOPS alone.
MLA (Multi-head Latent Attention)Attention variant that stores one small compressed vector per token in the KV cache instead of the full keys and values, introduced by DeepSeek-V2.
MoEMixture of Experts โ€” architecture activating only some sub-networks per token, enabling huge models with controlled inference cost.
Multi-GPUArchitecture using several GPUs in one machine or a cluster to increase memory and throughput; the gains depend on the interconnect (PCIe, NVLink).
NCCLNVIDIA collective communication library optimized for large-scale GPU-to-GPU transfers.
NPUNeural Processing Unit โ€” AI accelerator in modern SoCs, useful for some tasks but limited for large LLMs.
NVLinkDedicated hardware link that connects NVIDIA GPUs at very high speed, bypassing the PCIe bus; it is gone from recent workstation and consumer cards.
NVSwitchNVIDIA switch that connects multiple GPUs into a fully non-blocking NVLink fabric inside a node.
OffloadingTechnique that places part of the model in RAM or on SSD when VRAM is insufficient, at the cost of reduced throughput.
OllamaSimplified local runtime to download and run LLMs via llama.cpp or MLX, with OpenAI-compatible API.
On-Premise (AI)AI infrastructure hosted and operated on the organizationโ€™s own equipment, without delegating to a cloud provider.
PagedAttentionKV Cache management technique using virtual memory blocks, popularized by vLLM.
PCIeStandard bus connecting CPU, GPU, SSD and other peripherals; it limits CPU-to-GPU offloading and Tensor Parallelism on multi-GPU setups without NVLink.
PFCPriority Flow Control โ€” Ethernet per-priority pause mechanism for a lossless network required by RoCE.
Pipeline ParallelismStrategy for distributing an LLM in layer slices across multiple machines.
PrefillInference phase that processes the initial prompt in parallel before word-by-word generation.
Prompt injectionAttack where untrusted content in the LLM context hijacks system instructions to exfiltrate data or execute unauthorized actions.
Q4 QuantizationThe most widely used 4-bit quantization format for local inference โ€” especially Q4_K_M in the GGUF/Ollama ecosystem.
QuantizationReducing numerical precision of LLM weights to lower memory footprint and speed up inference.
RAGArchitecture that combines document retrieval and LLM generation to ground answers in internal sources.
RAGASFramework that evaluates a RAG system by separating retrieval quality from answer faithfulness, to diagnose where the pipeline fails.
RAMSystem main memory, second choice for LLM inference when VRAM is insufficient.
RayDistributed computing framework for multi-node LLM orchestration in production.
RDMANetwork technique that reads and writes remote memory without going through the CPU, with low latency; it comes in InfiniBand and RoCE implementations.
RoCEProtocol bringing RDMA benefits to standard Ethernet, without a dedicated InfiniBand fabric, at the cost of configuring a lossless network (PFC, ECN).
SmolAgentsLightweight Hugging Face framework for local agentic orchestrationโ€”a sovereign alternative to LangChain.
SoCSystem on a Chip: a single chip that combines processor, graphics, AI accelerator and memory controller; the basis of Apple Silicon Macs and DGX Spark.
Sparse attentionAttention where each token rereads only a selection of past tokens judged relevant, instead of the whole history, to cut the cost of long contexts.
Speculative DecodingInference acceleration technique where a small fast model proposes tokens that the large model verifies in a single pass; both models must be loaded.
Tensor ParallelismStrategy for distributing an LLM by splitting weight matrices across multiple GPUs on the same node.
TensorRT-LLMNVIDIA SDK for optimized LLM inference (native PyTorch, FP8/NVFP4) on datacenter GPUs.
TFLOPSUnit measuring floating-point compute capacity per second; useful to compare raw compute, but not enough on its own to predict LLM performance.
ThunderboltHigh-throughput wired interface for workstations and desktop AI clusters.
TokenizationSplitting text into numeric units (tokens) before an LLM processes it.
Tokens per secondNumber of tokens generated per second during generation; a smoothness indicator, to be read alongside TTFT given the model, engine and hardware.
Tool calling (Function / Tool Calling)An LLMโ€™s ability to emit structured requests to external functions (API, SQL, code) rather than free-form text.
TTFTTime To First Token: time between sending a request and receiving the first response token, which mostly reflects prefill and complements the tokens/s metric.
Unified MemoryArchitecture where CPU, GPU and sometimes NPU share a single memory pool, avoiding some copies over PCIe; found on Apple Silicon and some recent AMD APUs.
Vector databaseDatabase specialized in storing and searching embedding vectors for RAG.
vLLMOpen-source high-throughput LLM inference engine for NVIDIA/AMD GPUs, multi-user production standard.
VRAMVery high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading.

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue โ†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.