Tokenization
π Short definition
Turning text into a sequence of numeric identifiers (tokens) the model can process, using a vocabulary learned during training.
π Detailed definition
Text is split into tokens (words, subwords, or characters) with an algorithm learned during trainingβthe most common being BPE (Byte Pair Encoding). Each token gets a unique number in the model vocabulary. That number is what the model actually sees.
Example: βWhere is Paris?β β [4502, 381, 1920, 30]
Vocabulary size varies by model: Llama 3 uses 128,256 tokens. A rare word may split into several subwords, increasing the token count consumed.
π‘ Why it matters for on-prem AI
Token count for a text directly sets KV Cache size in VRAM and the cost of Prefill. A 200-page PDF can represent hundreds of thousands of tokens.
β οΈ Common pitfalls
- The same text yields different token counts per model tokenizerβdo not assume β1 page β 500 tokensβ is universal.
- French often uses more tokens than English (more subwords created).
π Go deeper
- π§ The Journey of a Prompt (Step 1: tokenization in action)
- πΎ KV Cache & Context (why token count affects VRAM)
π See also
Memory that keeps the attention keys and values already computed to avoid recomputing them, at the cost of memory use that grows with context length.Very high-bandwidth memory attached to the GPU, storing weights, KV Cache and buffers; its capacity and bandwidth determine which models run without offloading.Inference phase that processes the initial prompt in parallel before word-by-word generation.Dense numerical representation of a token or document in vector space.Maximum number of tokens an LLM can handle in active input β drives dynamic inference memory cost.