Skip to content

πŸ“– AI Glossary

This glossary centralizes key on-prem AI concepts.
Use it as a quick entry point, then open detailed entries.

Full index: all lexicon entries (alphabetical list, updated at build).


🚢 Beginner β€” getting started

Not sure where to begin:

LLM β†’ LLM inference β†’ 🧠 The Journey of a Prompt β†’ Prefill / Decoding β†’ Memory Wall β†’ 🏎️ Memory Bandwidth



Acronym index

AcronymMeaning
APUAccelerated Processing Unit
CUDACompute Unified Device Architecture (NVIDIA GPU platform)
ECNExplicit Congestion Notification
HBMHigh Bandwidth Memory
IBInfiniBand
KVKey-Value
LLMLarge Language Model
LPDDR5XLow Power Double Data Rate 5X (low-power memory)
MLAMulti-head Latent Attention
MoEMixture of Experts
NCCLNVIDIA Collective Communications Library
NPUNeural Processing Unit
PCIePeripheral Component Interconnect Express
PFCPriority Flow Control
Q44-bit quantization
RAGRetrieval-Augmented Generation
RAMRandom Access Memory
RDMARemote Direct Memory Access
RoCERDMA over Converged Ethernet
SoCSystem on a Chip
TFLOPSTera Floating Point Operations Per Second
TTFTTime To First Token
VRAMVideo RAM

LLM foundations

TermQuick definition
TokenisationSplitting text into numeric units the model processes.
EmbeddingCoordinate vector encoding a token’s meaning.
AttentionMechanism weighting each token’s importance in context.

Memory & performance

TermQuick definition
Memory WallPerformance limit from memory transfers rather than compute.
Memory bandwidthMemory data transfer rate, often in GB/s.
PrefillPrompt ingestion phase, more parallel.
DecodingAutoregressive token-by-token phase, often memory-bound.
KV CacheCached attention keys/values to speed generation.
MLAAttention that stores a compressed summary per token instead of full keys/values.
Sparse attentionEach token rereads only a selection of relevant past tokens.
Linear attentionContext memory kept in a fixed-size state instead of a growing cache.
Tokens/sGeneration throughput in tokens per second.

Software stack

TermQuick definition
OllamaLocal llama.cpp runtime + OpenAI API for testing and lab use.
vLLMHigh-throughput production engine, PagedAttention, multi-user.
SGLangAgentic inference, RadixAttention, structured JSON.
RadixAttentionPrefix-tree KV Cache (SGLang).
Speculative DecodingSmall draft model + large validator model to speed decoding.
PagedAttentionKV Cache managed in virtual blocks; key to vLLM production.
GGUFPortable format for local inference with llama.cpp/Ollama.
TensorRT-LLMNVIDIA SDK for highly optimized datacenter GPU compilation.
ExoP2P orchestrator for desktop AI clusters (Mac Mini + Thunderbolt).
RayDistributed framework for multi-node production deployment.
Pipeline ParallelismSplitting a model into layer slices across nodes.
Tensor ParallelismSplitting matrices within one NVLink multi-GPU node.
ThunderboltDesktop interconnect up to 80 Gb/s for Exo clusters.

Infrastructure & architecture

TermQuick definition
Unified memoryShared CPU/GPU/NPU memory pool in an SoC.
OffloadingPartial move of weights/activations across memory tiers by capacity.
Multi-GPUSeveral GPUs for more capacity and/or throughput.
NVLinkHigh-bandwidth GPU interconnect (pro/datacenter tiers).
NVSwitchNon-blocking NVLink fabric among all GPUs in an HGX node.
PCIeStandard CPU↔GPU bus; inter-GPU bottleneck without NVLink.

Networking & clustering

TermQuick definition
RDMAMemory-to-memory transfer without CPU copy for multi-node clusters.
RoCERDMA over Ethernet; needs PFC + ECN for lossless operation.
InfiniBandDedicated HPC/AI network fabric, natively lossless, ~400–800 Gb/s.
GPUDirect RDMADirect VRAM↔NIC transfer without CPU copy.
NCCLNVIDIA library for GPU-to-GPU collective communication.
PFCPriority Flow Control β€” per-priority pause for lossless RoCE.
ECNExplicit Congestion Notification β€” proactive congestion handling.

Foundational concepts

TermQuick definition
On-PremiseAI infrastructure hosted on the organization’s own hardware.
LLMLarge language model trained on vast text corpora.
LLM inferenceRunning a trained LLM to generate text on demand.

Applied AI

TermQuick definition
LLM inferenceRunning a trained LLM to generate text on demand.
RAGArchitecture enriching LLM context from a document base.
Vector databaseStore and search documents by embedding similarity.
GraphRAGRAG over a knowledge graph for global queries.
Memory TreeHierarchical memory loading only useful summaries and nodes.
Autonomous agentLLM with tools that loops to complete tasks.
Custodian agentMaintenance agent proposing corrections awaiting human validation.
Human-in-the-loopGovernance where humans validate critical actions before apply.
Excessive AgencyOWASP LLM06: agent too permissive in tools, rights, or autonomy.
LiteLLMOpenAI-compatible gateway routing to local or cloud models.
SmolAgentsLightweight Hugging Face agent framework without cloud telemetry.
LangGraphState graph for multi-step agents and agentic RAG.
Tool callingFunction / tool calling β€” LLM to structured external functions.
Prompt injectionOWASP LLM01 β€” hijacking via untrusted content (RAG, web).
Zero Data RetentionZDR clause: no cloud persistence of prompts.
Multi-tenantMulti-organization isolation; inter-tenant RAG risk.
Context windowMaximum tokens the model can handle in active input.
QuantizationLower numeric precision to cut memory and compute cost.
MoEArchitecture with specialized experts partially activated per token.

Model evaluation

TermQuick definition
LLM benchmarkStandardized tests to compare language models.
LLM-as-a-judgeUsing an LLM as judge to score or compare answers.
RAGASFramework evaluating RAG pipelines: retrieval, faithfulness, relevance.

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.